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2  Introduction 

This  project  is  aimed  at  improving  the  state  of  the  art  of  image-guided  and  minimally 
invasive  procedures  by  developing  a  new  generation  of  clinical  techniques  along  with  the 
computer-based  hardware  and  software  needed  for  their  implementation.  The  focus  of  the 
project  is  to  develop  physician  assist  systems  incorporating  robotics,  tracking,  and 
visualization  to  improve  the  precision  of  instrument  placement  and  manipulation  in 
minimally  invasive  procedures.  The  project  is  led  by  the  Imaging  Sciences  and 
Information  Systems  (ISIS)  Center  of  the  Department  of  Radiology  at  Georgetown 
University.  Project  collaborators  include  the  Department  of  Radiation  Medicine  at 
Georgetown,  the  Urology  Robotics  Group  at  Johns  Hopkins  Medical  Institutions,  the 
NSF  sponsored  Engineering  Research  Center  for  Computer  Integrated  Surgical  Systems 
and  Technology  at  Johns  Hopkins  University,  and  the  Engineering  School  at  the  Catholic 
University  of  America,  as  well  as  several  commercial  partners. 


3  Report  Body 

This  section  describes  the  research  accomplishments  associated  with  each  task  in  the 
statement  of  work.  This  is  the  final  report  and  includes  research  performed  from  January 
15,  1999  -  February  1,  2005.  Since  annual  reports  have  been  provided  every  year  and 
included  detailed  progress  updates,  this  report  will  focus  on  the  work  done  over  the  past 
year  from  February  1,  2004  to  February  1,  2005  (although  reference  will  also  be  made  to 
previous  work  in  some  places).  The  award  number  is  DAMD17-99-1-9022. 

3.1  Task  1:  Program  Planning  and  Management 

Throughout  the  award  period,  program  planning  and  management  focuses  on  maintaining 
project  direction  as  well  as  developing  strong  partnerships  with  project  collaborators. 
Project  planning  and  review  meetings  are  held  monthly  at  the  ISIS  Center.  In  the  2004 
calendar  year  major  progress  was  made  on  several  projects  including  fluoroscopy 
servoing  for  the  needle  driver  robot,  electromagnetic  tracking  of  guidewires  for  image 
guidance,  rehabilitation  robotics,  and  three-dimensional  visualization.  We  also  organized 
an  extremely  successful  workshop  titled  “OR2020:  The  Operating  Room  of  the  Future”. 

The  group  continues  to  be  very  active  in  presenting  their  work  at  national  and 
international  conferences.  Dr.  Cleary  is  now  the  co-chair  for  the  “Image-guided 
Procedures,  Visualization,  and  Display”  track  of  the  SPIE  Medical  Imaging  meeting, 
which  is  held  every  February  in  San  Diego.  The  group  demonstrated  several  of  their 
prototypes  at  the  Computer  Aided  Radiology  and  Surgery  Conference  (CARS)  in  June 
2004  in  Chicago.  Dr.  Cleary  is  also  currently  organizing  medical  robotics  workshops  for 
the  April  2005  IEEE  Robotics  and  Automation  Conference  and  the  June  2005  CARS 
conference. 
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3.2  Task  2:  Spinal  Robotics 

One  of  the  key  outcomes  of  this  project  has  been  the  development  of  a  needle  driver 
robot  in  collaboration  with  the  Urology  Robotics  Program  at  Johns  Hopkins  University 
(Dan  Stoianovici,  PhD).  Precision  placement  of  instruments  such  as  needles  could  be 
enhanced  by  medical  robotics  and  we  have  become  an  international  leader  in  this  area. 

As  detailed  in  the  last  annual  report,  the  initial  clinical  trial  of  20  patients  using  the 
needle  driver  robot  for  spinal  blocks  was  conducted  at  Georgetown  University  from 
August  2002  to  December  2002.  A  picture  of  the  robot  is  shown  in  Figure  1  and  a  picture 
of  the  first  patient  is  shown  in  Figure  2.  The  clinical  trial  showed  that  is  feasible  to  use  a 
joystick  controlled  robot  for  nerve  and  facet  blocks.  Based  on  the  success  of  this  initial 
trial,  we  applied  to  the  FDA  to  continue  the  clinical  trial  with  an  additional  80  patients. 
We  have  now  received  FDA  approval  for  as  well  as  approval  from  the  Georgetown  and 
Army  IRB  boards.  We  have  completed  a  few  new  patients  under  this  revised  protocol, 
but  have  not  yet  gathered  enough  new  data  to  report  any  more  conclusions. 


Figure  1:  Robotic  device  showing  mechanical  arm  and  joystick  control 

(courtesy  of  Dan  Stoianovici,  PhD,  Johns  Hopkins  Urology  Robotics) 


Figure  2:  Clinical  trial  of  robotic  device  for  nerve  and  facet  blocks  at  Georgetown 
University  (interventional  radiologist  is  Vance  Watson,  MD) 
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This  work  is  also  being  continued  through  two  additional  tasks  which  will  now  be 
described: 

Section  5.2.1:  fluoroscopy  servoing  for  needle  alignment 
Section  5.2.2:  robotically  assisted  lung  biopsy 

3.2.1  Fluoroscopy  Servoing  for  Needle  Alignment 

In  the  clinical  trial  described  in  Section  5.2,  the  first  step  in  placing  the  needle  using  the 
robot  is  to  align  the  needle  toward  the  target  to  be  reached.  By  capturing  the  fluoroscopy 
images  on  the  computer  and  rapidly  processing  them,  it  is  possible  to  automatically  align 
the  needle  for  the  physician.  This  automatic  needle  alignment  is  a  first  step  towards 
further  automating  the  procedure  and  an  initial  step  towards  the  completely  automatic 
biopsy  systems  of  the  future.  The  alignment  technique  is  “fluoroscopy  servoing”  and  has 
been  explored  by  just  a  few  researchers. 


Our  recent  work  in  this  area  is  detailed  in  [Mihaescu  2005]  and  just  a  summary  will  be 
given  here.  The  whole  process  consists  of  two  phases:  translation  and  alignment.  The 
translation  process  will  place  the  tip  of  the  needle  on  the  line  of  sight  that  passes  through 
the  target.  The  alignment  stage  will  orient  the  needle  until  it  is  parallel  to  the  line  of  sight 
between  the  anatomical  target  and  the  projection  of  the  target  on  the  fluoroscopic  image. 
The  positions  of  the  needle  tip  in  the  image  plane  are  found  by  segmenting  the  needle 
from  the  fluoroscopic  images.  A  typical  fluoroscopic  image  is  presented  in  Figure  3, 
where  the  needle  appears  to  overlap  a  section  of  the  patient’s  spine. 


Figure  3.  Fluoroscopic  input  image 


Figure  4.  Flow  diagram  of  the  segmentation 
method  and  needle  tip  position  estimation 
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The  process  used  for  segmenting  the  needle  is  illustrated  in  Figure  4.  The  essential  aspect 
of  this  segmentation  process  is  the  detection  of  intensity  ridge  points  on  the  fluoroscopic 
image.  For  details  please  refer  to  the  paper  in  the  appendix  [Mihaescu  2005]. 


3.2.2  Robotically  Assisted  Lung  Biopsy 

Based  on  our  initial  work  in  needle  placement  for  spinal  procedures  using  x-ray 
fluoroscopy,  we  expanded  into  needle  placement  for  lung  biopsy  under  CT  fluoroscopy. 
This  work  has  also  been  partially  supported  by  an  NCI/NIH  grant. 

The  first  phase  of  this  project  has  been  completed  and  a  paper  has  been  submitted  to 
Academic  Radiology  [Cleary  2004].  The  goal  of  this  phase  was  to  evaluate  using  a 
joystick-controlled  robotic  needle  driver  for  lung  biopsy  in  a  phantom  study. 

A  respiratory  motion  phantom  incorporating  cranial-caudal  motion  of  a  foam  liver 
phantom  was  developed.  Synthetic  lesions  of  about  1 .5  mm  in  diameter  were  created 
using  an  agar  mix.  CT  fluoroscopy  was  used  to  visualize  the  motion  of  a  lesion  in  and  out 
of  the  scan  plane.  The  phantom  could  be  paused  on  demand  for  up  to  30  seconds  to 
simulate  a  breath  hold.  A  joystick-controlled  robotic  needle  driver  was  used  by  a 
radiologist  experienced  in  lung  biopsy  to  drive  an  1 8  gauge  needle  on  the  scan  plane 
toward  the  lesion.  The  experimental  setup  is  shown  in  Figure  5. 


Figure  5:  Phantom  study  of  robotically  assisted  lung  biopsy  under  CT  fluoroscopy  at 
Georgetown  University  (interventional  radiologist  is  Charles  White,  MD) 


Page  8 


Periscopic  Spine  Surgery 


Final  Report:  15  Jan  99-01  Feb  05 


The  radiologist  was  able  to  hit  the  lesion  on  all  20  attempts.  A  CT  image  of  the  needle  in 
the  lesion  is  shown  in  Figure  6.  The  average  time  to  drive  the  needle  was  12.1  seconds 
with  a  standard  deviation  of  3.1  seconds.  The  average  tube  current-time  product  was  955 
mAs  with  a  standard  deviation  of  1 80  mAs.  The  average  dose-length  product  was  92.5 
mGycm  with  a  standard  deviation  of  42  mGycm. 

The  conclusion  was  that  a  joystick-controlled  robotic  needle  driver  can  be  used  to 
accurately  target  lesions  in  a  respiring  phantom  under  CT  fluoroscopy.  This  technique 
may  allow  the  radiologist  to  more  precisely  manipulate  the  needle  with  much  less 
radiation  exposure.  Additional  studies  are  planned  to  further  automate  this  process  and  to 
investigate  the  role  of  robotic  systems  for  lung  biopsy  in  the  clinical  area. 
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Figure  6:  Axial  CT  image  of  needle  in  lung  lesion 


3.3  Task  3:  Liver  Respiratory  Motion  Simulator  and 
Internal  Organ  Tracking 

The  goal  of  this  task  is  to  investigate  the  use  of  electromagnetic  tracking  for  precisely 
locating  internal  organs  such  as  the  liver  during  interventional  procedures.  This  is  an 
ongoing  collaboration  with  Northern  Digital  (Waterloo,  Canada)  and  Traxtal 
Technologies  (Bellaire,  Texas).  Northern  Digital  has  developed  the  AURORA™ 
electromagnetic  tracking  system,  which  enables  instruments  that  are  fitted  with  a  sensing 
coil  to  be  tracked  and  overlaid  on  an  image  of  the  anatomy.  Our  research  group  at 
Georgetown  served  as  a  beta  test  site  and  was  one  of  the  first  research  groups  worldwide 
to  receive  this  equipment.  The  system  has  recently  been  released  as  a  commercial  product 
as  shown  in  Figure  7. 
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Figure  7:  AURORA™  sensors,  electromagnetic  tracking  system  components, 

and  measurement  volume 

The  left  picture  shows  (from  left  to  right)  the  control  unit,  sensor  interface  device,  and  electromagnetic 
field  generator.  The  middle  picture  shows  the  sensor  coils  along  with  the  electrical  wires  protruding  from 
the  coil,  compared  to  a  match.  The  right  picture  shows  the  measurement  volume  in  mm  relative  to  the 
location  of  the  field  generator.  (Photos  courtesy  of  Northern  Digital,  Inc.) 

As  part  of  this  research,  we  have  developed  an  image-guided  surgery  system 
incorporating  electromagnetic  tracking.  Figure  8  shows  the  tracking  system  as  recently 
used  in  an  approved  swine  study  in  the  interventional  suite.  Electromagnetic  tracking  of 
instruments  in  the  interventional  suite  enables  image  overlay  of  preoperative  CT  scans 
and  provides  multimodality  guidance  while  decreasing  x-ray  dose.  In  the  study  described 
here,  an  electromagnetically  tracked  guidewire  was  manipulated  in  a  swine  model  with 
co-display  of  CT  and  angiographic  images  [Wood  2004]. 


Figure  8:  Swine  study  in  the  interventional  suite 
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An  enhanced  CT  scan  of  a  swine  chest  and  abdomen  was  obtained.  A  rigid  body 
registration  was  carried  out  by  identifying  the  points  on  the  CT  scan  and  in 
electromagnetic  space.  The  guidewire  was  then  initially  moved  in  free  air  over  the 
abdominal  area  of  the  swine  to  verify  that  the  registration  software  was  working  and  that 
is  could  be  tracked. 

The  internal  jugular  vein  and  right  carotid  artery  were  percutaneously  accessed  using 
ultrasound  guidance  and  modified  Seldinger  technique  with  a  21  gauge  needle  and  an 
.018  inch  guidewire.  As  the  guidewire  was  advanced  down  the  aorta,  the  corresponding 
axial  CT  scan  was  displayed  simultaneously  with  the  real-time  angiographic  x-ray  image. 
The  system  provided  good  qualitative  agreement  in  the  opinion  of  the  experienced 
interventional  radiologist  manipulating  the  guidewire.  As  the  guidewire  was  advanced 
into  the  selected  vessel,  the  image-guided  system  was  able  to  actively  track  the  guidewire 
and  display  axial,  coronal,  and  sagittal  CT  scans  at  the  same  level,  using  the 
electromagnetically  tracked  sensor  position  to  select  the  slice  location.  The  resulting 
display  is  shown  in  Figure  9. 
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Figure  9:  Axial,  sagittal,  and  coronal  image  overlay  of 
electromagnetically  tracked  guidewire 
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3.4  Task  4:  Radiation  Treatment  (Stereotactic  Radiosurgery) 

Organs  in  the  abdomen  and  thorax  move  and  deform  during  respiration,  and  these 
changes  place  a  severe  limitation  on  the  effectiveness  of  radiation  medicine  in  the  lungs, 
liver,  and  pancreas.  Georgetown  University  Medical  Center  is  an  emerging  leader  in  a 
new  form  of  radiosurgery  for  these  types  of  tumors.  These  treatments  are  enabled  by  two 
key  technologies:  (1)  the  Cyberknife,  a  precision  radiosurgery  machine  based  around  a 
compact  linac  and  a  robotic  positioning  arm,  and  (2)  Synchrony,  a  system  for  optically 
tracking  respiratory  motion  and  steering  the  radiation  beam  to  follow  a  moving  tumor. 
This  research  is  being  carried  out  in  conjunction  with  the  Radiation  Medicine  Department 
at  Medstar  Georgetown  University  Hospital. 


Figure  10:  Photograph  of  Cyberknife  Stereotactic  Radiosurgery  System 
(Accuray,  Sunnyvale,  CA) 


A  key  development  in  the  past  year  has  been  the  refinement  of  our  motion  simulator 
(Figure  11,  Figure  12)  and  our  continuing  efforts  to  develop  realistic  respiratory  motion 
models.  The  simulator  was  redesigned  to  make  it  more  robust  and  to  match  a  change  in 
the  Cyberknife  system  geometry  [Zhou  2004].  We  have  also  refined  the  control  software 
to  provide  a  graphical  user  interface  for  the  program  and  the  ability  to  graphically 
preview  motion  files.  This  work  builds  on  our  previous  efforts  to  measure  and 
characterize  respiratory  motion.  The  motion  simulator  is  capable  of  playing  back  almost 
any  type  of  respiratory  motion,  from  simple  sinusoidal  movements  to  irregular  patterns 
based  on  recorded  human  data.  Thus,  with  this  simulator,  we  can  ‘play  back’  an 
individual  patient’s  breathing  pattern  and  use  it  as  a  test  bed  for  different  motion 
compensation  strategies.  Furthermore,  because  it  has  two  independent  platforms,  it  can 
produce  skin  motion  that  is  different  from  internal  organ  motion,  which  is  a  more  realistic 
simulation  that  could  be  produced  by  a  single  platform  simulator. 
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Figure  11:  Detail  photo  of  the  respiratory 
motion  simulator.  Stage  on  the  left 
simulates  internal  organ  motion,  stage  on 
the  right  simulates  skin  motion _ 


Figure  12:  Respiratory  motion  simulator  in 
use  during  Cyberknife  experiment. 
Synchrony  optical  beacons  are  attached  to  the 
_ skin  motion  stage  for  tracking 


Patient  respiratory  motion  patterns  are  drawn  from  several  sources.  One  of  these  is 
magnetic  resonance  imaging  (MRI).  Using  dynamic  MRI,  we  are  able  to  capture 
snapshots  of  the  anatomy  during  respiration  (Figure  13)  and  use  these  to  build  a 
numerical  model  of  how  internal  organs  move.  We  use  both  coronal  and  axial  imaging 
slices,  because  the  coronal  orientation  more  clearly  shows  internal  organ  motion  (which 
is  largely  along  the  superior-inferior  axis)  and  the  axial  slices  more  clearly  show  the  skin 
motion.  MRI  therefore  allows  us  to  determine  for  an  individual  patient  how  the  skin 
motion  translates  to  the  motion  of  an  interna!  organ;  these  data  can  then  be  used  as  input 
for  the  motion  simulator  [Wong  2004a,  2004b,  2005]. 


Figure  13:  Dynamic  MRI  images  of  patients  acquired  during  free  breathing.  The  imaging 
sequence  used  is  trueFISP,  which  is  fast  enough  to  “freeze”  the  respiratory  motion. 


Another  method  for  creating  motion  simulator  patterns  is  to  examine  the  treatment  logs 
from  patients  undergoing  standard  Cyberknife/Synchrony  treatments.  During  Synchrony 
treatments,  patients  wear  a  tight  fitting  fabric  vest  to  which  small  LEDs  can  be  attached. 
These  LEDs  are  observed  by  a  Flashpoint  optical  tracker  (Boulder  Innovation  Group, 
Boulder,  CO,  USA)  which  measures  their  3D  position  several  times  per  second.  The 
patient  is  also  periodically  imaged  with  a  stereo  pair  of  x-ray  tubes  and  detectors.  Image 
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analysis  software  automatically  segments  the  x-ray  images  and  extracts  the  location  of 
internal  metallic  fiducials  that  have  been  percutaneously  implanted  in  the  tissue  around 
the  tumor  or  in  a  nearby  bony  structure.  The  fiducials  are  placed  before  treatment 
planning  and  thus  have  a  known  geometric  relationship  to  the  center  of  the  treatment 
target.  The  combination  of  these  data  therefore  provide  simultaneous  measurement  of  an 
internal  target  position  and  an  external  skin  position. 

We  are  also  investigating  the  use  of  electromagnetic  tracking  in  the  Cyberknife 
environment.  This  technology  has  the  advantage  of  being  able  to  provide  direct  motion 
tracking  of  internal  organs,  thereby  eliminating  potential  uncertainties  associated  with 
skin  motion  tracking.  The  test  was  conducted  in  the  CyberKnife  suite  at  Georgetown  as 
shown  in  Figure  14.  The  equipment  consisted  of  the  Aurora  system,  the  CyberKnife,  and 
the  needle  driver  robot,  used  here  as  a  precise  positioner.  The  needle  driver  robot  is 
holding  an  18  gauge  needle  that  has  an  Aurora  sensor  inside  its  tip.  With  some 
modifications,  these  needles  might  be  used  for  tumor  tracking  in  the  abdomen  or  could  be 
repackaged  into  other  forms. 

Test  data  was  gathered  by  moving  the  robot  in  a  pre-specified  pattern,  which  was  a 
rectangular  volume  of  200  by  40  by  50  mm.  A  total  of  34  points  were  defined  within  this 
volume.  At  each  measurement  point  the  robot  stops  and  the  control  computer  calls  the 
Aurora  system  to  gather  100  readings  which  are  then  stored  in  a  file.  Aurora  position 
data  were  collected  with  the  CyberKnife  LINAC  at  two  different  distances  -  1 000  mm 
and  500  mm  from  the  tracking  sensors  -  and  with  the  LINAC  beam  on  and  off  at  each 
distance,  for  a  total  of  four  data  sets. 


Figure  14:  Test  setup  showing  CyberKnife,  Aurora  system,  and  robot  positioner 
holding  electromagnetically  tracked  needle. 

As  can  be  seen  in  the  table  below,  the  average  error  was  between  1.1  and  1 .2  mm  for  all 
four  tests  with  maximum  errors  of  4.1  to  4.3  mm.  Based  on  this  data,  there  appears  to  be 
no  significant  effect  on  Aurora  accuracy  from  turning  the  CyberKnife  beam  on.  The 
system  error  also  appears  to  be  acceptable  for  clinical  use.  However,  this  would  need  to 
be  further  validated  in  future  studies. 
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Test 

Set 

Beam 

Distance 
From  Linac 

Minimum 

Error 

Maximum 

Error 

Average 

Error 

Standard 

Deviation 

1 

OFF 

1000 

0.0007 

4.2966 

1.1501 

0.8762 

2 

ON 

1000 

0.0049 

4.1068 

1.1031 

0.8298 

3 

ON 

500 

0.0005 

4.1683 

1.1876 

0.8698 

4 

OFF 

500 

0.0010 

4.1458 

1.1757 

0.8710 

Table  1:  Error  and  standard  deviation  of  measurements  (all  numbers  in  mm) 


The  primary  disadvantage  of  this  technology  is  that  the  markers  cannot  yet  be  made 
wireless,  so  patients  would  need  to  have  wires  emerging  from  their  skin  in  order  to  read 
the  data  from  the  tracked  fiducial. 


3.5  Task  5:  Rehabilitation  Robotics 

The  primary  goal  of  this  effort  is  to  develop  robotic  systems  for  assessment  and  physical 
therapy.  The  secondary  goal  is  to  expand  this  capability  to  the  internet.  We  currently 
have  seven  ongoing  projects  in  rehabilitation  robotics  spanning  three  robotic  testbeds 
which  are  each  described  briefly  below.  We  will  continue  to  expand  our  activities  in  this 
area  through  ongoing  collaborations  with  the  National  Rehabilitation  Flospital,  the 
University  of  Maryland,  the  Center  of  Excellence  for  Remote  and  Medically  Under 
Served  Areas  (CERMUSA),  and  Interactive  Motion  Technologies.  As  some  of  these 
devices  may  be  controlled  through  the  Internet,  a  systematic  study  of  information 
assurance  and  vulnerabilities  will  be  an  integral  part  of  this  effort. 


3.5.1  lnMotion2  Robot 

The  ISIS  Center  operaties  a  telerehabilitation  testbed  consisting  of  a  pair  of  InMotion2 
(IM2)  robots  from  Interactive  Motion  Technology,  Inc.  (see  Figure  15).  The  specific 
aims  of  our  research  are  twofold:  to  enable  a  clinician  to  assess  the  physical  condition  of 
a  patient's  arm  using  metrics  such  as  strength,  dexterity,  range  of  motion,  and  spasticity; 
and  to  help  a  clinician  perform  cooperative  rehabilitative  tasks  with  a  patient  using  a 
virtual  environment  intended  to  simulate  active  daily  living  (ADL)  tasks. 

The  IM2  Robot  has  direct-drive  motors  at  the  base  which  transmit  forces  to  the  handle 
through  a  four-bar  linkage  as  shown  in  see  Figure  16.  The  handle  moves  within  a 
rectangle  of  dimension  90  x  60  cm  just  above  the  surface  of  the  table.  The  maximum 
continuous  force  output  of  the  robot  at  the  handle  is  approximately  30  N  in  each 
direction.  The  handle  is  pinned  to  the  end  of  the  last  link  providing  a  third,  unactuated 
degree  of  freedom.  The  IM2  robot  is  FDA-approved  as  both  an  evaluation  device  and 
therapy  device  with  device  listings  R043555  and  B004584,  respectively. 

We  are  currently  using  the  IM2  testbed  for  four  different  projects,  all  of  which  will 
culminate  in  clinical  trials  to  be  conducted  over  the  internet.  Two  projects  are 
specifically  aimed  at  developing  protocols  for  stroke  therapy.  One  project  is  to  perform 


Page  15 


Periscopic  Spine  Surgery 


Final  Report:  15  Jan  99  -  01  Feb  05 


neuroscience  experiments  for  Parkinsonian  patients  over  the  internet.  The  remaining 
project  is  to  develop  cooperative,  functional  rehabilitation  for  populations  with 
neurological  deficits  by  using  virtual  reality  technologies.  These  projects  are  described  in 
more  detail  below  [Carignan  2004a,  2004b,  2004c]. 


Figure  15:  An  InMotion2  Robot  undergoing 
systems  checkout  during  installation 


Figure  16:  Top  view  of  InMotion2 
workstation  showing  4-bar  linkage,  direct 
drive  motors,  and  flat  panel  display 


3.5. 1.1  Stroke  Rehabilitation 

We  are  working  with  the  National  Rehabilitation  Hospital  and  Morgan  State  University 
to  develop  robotic  protocols  for  stroke  therapy.  The  case  study  we  are  developing  is  to 
convert  “skateboard”  therapy  currently  being  done  manually  into  a  protocol  for  the  IM2 
robot.  When  the  therapy  is  conducted  manually,  the  patient  rests  their  elbow  on  a  cart 
called  a  "skateboard"  which  moves  along  a  table  top  as  their  hand  moves  in  straight  line 
directions.  Since  the  IM2  is  a  planar,  tabletop  device,  this  was  an  ideal  candidate  for 
transfer  to  robotic  therapy. 

Our  eventual  goal  is  to  conduct  a  pilot  study  to  treat  NRH  patients  over  the  internet  using 
stroke  doctor  at  Georgetown.  The  prototype  software  has  been  developed,  but  is 
currently  being  modified  to  provide  force  inputs  to  assist  the  patient  during  the  task.  The 
operator  interface  for  the  clinician  is  shown  in  Figure  17  and  a  sample  shape  traced  out 
by  the  patient  is  shown  in  Figure  18. 
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Figure  17:  Graphical  interface  used  by  Figure  18:  A  sample  figure  used  in  skateboard 
clinician  in  skateboard  therapy  protocol  therapy.  The  bold  line  is  the  path 

_ traced  out  by  the  patient. _ 


The  second  stroke  therapy  effort  we  are  engaged  in  is  with  the  John  P.  Murtha 
Neuroscience  and  Pain  Institute  in  Johnston,  PA.  We  are  working  with  personnel  from 
the  institute  and  CERMUSA  to  broaden  a  pilot  study  done  at  the  Helen  Hayes  Hospital 
which  varied  center  of  workspace  for  IM2  therapy  (see  Figure  19).  The  goal  is  to  enlarge 
database  and  give  CERMUSA  experience  with  robotic  therapy.  The  protocol  has  been 
written  and  the  IRB  application  is  currently  being  prepared. 

3.5.1.2  Sensorimotor  Adaptation 

This  neuroscience  investigation  is  a  joint  project  being  conducted  with  the  Department  of 
Kinesiology  at  the  University  of  Maryland.  It  is  a  pilot  study  to  examine  the  feasibility  of 
using  robotic  platforms  to  develop  large-scale  computational  models  of  sensorimotor 
function  of  healthy  subjects.  This  research  is  an  intermediate  step  towards  (a) 
understanding,  validating  and  testing  brain  model-machine  interfaces,  and  (b)  eventually 
developing  robotic  interventions  for  treating  patients  with  neural  disorders  such  as 
Parkinsons  Disease  and  stroke  hemiplagia. 

By  using  the  FDA-approved  InMotion2  robot  to  apply  either  kinematic  distortions  or 
small  force  perturbations  to  the  hand,  we  will  study  the  changes  in  the  kinematics  of 
movement  as  a  function  of  practice.  From  this  data,  large-scale  computer  models  of 
neural  circuits  can  be  developed  to  help  elucidate  not  only  the  biological  principles 
underlying  adaptive  cognitive-motor  behavior,  but  also  to  integrate  and  advance  our 
understanding  of  multi-modal  brain  data.  By  doing  this  study  with  healthy  subjects,  we 
hope  to  establish  a  baseline  model  which  can  then  be  compared  with  future  studies 
involving  patients  with  neural  disorders. 

The  protocol  for  this  trial  has  been  approved  by  both  Georgetown  University  and  the  U.S. 
Army.  Approval  by  the  University  of  Maryland  IRB  is  currently  pending.  The  basic 
setup  for  this  study  is  shown  in  Figure  20.  We  expect  the  trial  to  be  underway  shortly. 
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Figure  19:  InMotion2  robot  at  CERMUSA  to 
be  used  in  stroke  therapy  clinical  trial 


Figure  20:  Operator  interface  for  proposed 
sensorimotor  clinical  trial 


3.5.1.3  Tele-Rehabilitation 

The  tele-rehabilitation  testbed  is  being  used  to  study  the  feasibility  of  conducting  remote 
assessment  and  therapy  over  the  internet.  The  specific  aims  of  our  research  are  twofold: 
to  enable  a  clinician  to  assess  the  physical  condition  of  a  patient's  arm  using  metrics  such 
as  strength,  dexterity,  range  of  motion,  and  spasticity;  and  to  help  a  clinician  perform 
cooperative  rehabilitative  tasks  with  a  patient  using  a  virtual  environment  intended  to 
simulate  active  daily  living  (ADL)  tasks.  The  system  architecture  for  connecting  the  two 
robots  over  the  internet  is  shown  in  Figure  21. 

To-date,  we  have  successfully  conducted  a  cooperative  beam  task  between  Georgetown 
University  and  Interactive  Motion  Technology  in  Cambridge,  Massachusetts.  Internet 
delays  of  up  to  110  ms  were  observed  which  caused  borderline  stability  without 
compensation.  The  wave-variable  controller  was  found  to  be  robust  to  variations  in  time- 
delay,  and  performance  degraded  only  gradually  with  increasing  delay.  The  degradation 
manifests  itself  as  a  higher  perceived  inertia  at  the  handle  which  was  still  quite 
manageable  even  with  artificially  induced  delays  of  over  1  sec. 

Current  work  is  focused  on  generating  more  realistic  3D  tasks  created  using  Open 
Inventor  Toolkit  and  rendered  on  a  head  mounted  display  (see  Figure  22).  A  head  tracker 
is  used  to  track  orientation  of  the  subject's  head  for  changing  viewing.  Future  work  will 
focus  on  generating  additional  cooperative  tasks  over  the  internet  aimed  at  improving 
recovery  from  stroke  paralysis  and  other  neurological  disorders. 
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Figure  21:  System  architecture  for 
cooperative  rehabilitation  over  the  internet 


Figure  22:  Cooperative  beam  task  using  robot 
_ haptic  interface  and  VR  goggles _ 


3.5.2  MGA  Exoskeleton 

The  Maryland-Georgetown-Army  (MGA)  Exoskeleton  is  an  upper  arm  robotic 
exoskeleton  being  developed  for  assessment  and  rehabilitation  of  shoulder  pathology. 
There  are  five  degrees  of  freedom:  three  joints  for  shoulder  rotation,  one  for  elbow 
flexion/extension,  and  one  for  scapula  elevation/depression  (see  prototype  in  Figure  23). 
The  exoskeleton  will  be  used  to  assess  arm  strength,  speed,  and  range  of  motion  using 
onboard  sensors,  and  function  as  both  a  resistance  trainer  and  virtual  reality  tool  for 
rehabilitation.  Operation  of  the  device  will  be  monitored  by  a  computer-controlled  safety 
system  based  on  an  architecture  developed  for  a  NASA  robotic  flight  experiment. 

This  project  is  a  joint  venture  between  the  Georgetown  University  Imaging  Science  and 
Information  Systems  (ISIS)  Center  and  the  Space  Systems  Laboratory  (SSL).  The  robotic 
hardware  will  be  built,  tested,  and  integrated  by  the  SSL  robotics  group,  and  the  ISIS 
Center  will  lead  the  controls  effort  and  assist  with  the  kinematic  design,  electronics,  and 
operations.  The  hardware  for  the  exoskeleton  is  currently  in  fabrication  (see  CAD 
drawing  in  Figure  24),  and  assembly  and  integration  will  commence  in  the  spring. 
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3.5.3  Keiser  Arm  Curl  Machine 

We  are  currently  developing  a  powered  exercise  machine  that  can  be  used  for  physical 
therapy.  A  Keiser  arm  curl  machine  was  retrofitted  with  a  linear  actuator  (see  Figure  25), 
and  an  onboard  microprocessor  controls  the  motor  based  on  feedback  from  a  position  and 
force  sensor.  Resistance  laws  are  programmed  on  a  PC  and  an  admittance  controller 
outputs  commands  to  the  motor.  The  experimental  setup  is  shown  in  Figure  26. 
Preliminary  results  for  simulating  free  weight  and  viscoelastic  resistances  were 
encouraging.  Future  work  will  include  building  an  analog  filter  and  using  shielded  cable 
for  the  load  cell  to  help  suppress  noise  in  the  force  signal.  After  these  hardware  issues 
are  addressed,  we  will  begin  testing  new  resistance  laws  that  change  over  the  course  of  a 
repetition. 


Figure  26:  Keiser  Arm  Curl  250  retrofitted 
with  the  Smart  Motor  actuator. 


Figure  25:  Ultramotion  Smart  Motor  beside 


the  Keiser  pneumatic  piston. 


3.6  Task  6:  Surgical  Planning  and  3D  Visualization 

The  goal  of  this  task  is  to  develop  a  surgical  planning  capability  that  allows  integration  of 
innovative  surgical  techniques  such  as  image  processing  and  visualization  for 
neurosurgery,  vascular  surgery,  and  interventional  radiology  applications.  We  have  two 
major  thursts: 

1.  image-guided  transbronchial  biopsy  (clinical  lead,  Eric  Anderson,  MD, 
pulmonology) 

2.  skull-base  subtraction  (clinical  lead,  Dan  Nguyen,  MD,  neuroradiology). 

3.6.1  Image-guided  Transbronchial  Biopsy 

In  conventional  transbronchial  biopsy,  the  planning  is  very  simple.  Pulmonologists  look 
at  the  pre-operative  CT  scan,  specify  the  target  biopsy  location,  and  estimate  the  biopsy 
location.  During  the  operation,  pulmonologists  manipulate  the  bronchoscope  to  the 
estimated  biopsy  location  and  take  a  biopsy  sample.  However,  the  failure  rate  is  quite 
high  as  bronchoscope  video  is  the  only  tool  used  to  determine  the  biopsy  location.  By 
providing  transparent  virtual  bronchoscopic  images,  doctors  are  able  to  see  the  target 
lymph  nodes  through  the  mucosa  wall,  which  can  result  in  enhanced  biopsy  accuracy.  We 
are  currently  developing  a  system  to  provide  these  capability.  To  date,  we  have 
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developed  some  of  the  software  components  including  transparent  visualization, 
registration  and  tracking.  Figure  27  shows  the  components  and  workflow  for  this  system. 


1.  Pre-operative 


/Tracking  devices 
in  the  working  channel 


Bronchoscope 


4.  Registration  & 

5.  Tracking 


Localization  Board 


6.  Side-by-side  display 


Surgical  Planning 
with  Thin  Cuts 

2.  Segmentation  of  lymph  node 

3.  Biopsy 


reformatted  images, 
volume  rendered  images  and 
the  virtual  endoscopic  images 


Figure  27.  System  overview  of  image-guided  transbronchial  biopsy 


3.6.2  Skull-base  Subtraction 

The  goal  of  this  research  project  is  to  develop  an  optimal  skull  base  subtraction  method  to 
assist  neuroradiologists  in  interpreting  high-resolution  CT  angiography  images.  We 
hypothesize  that  combining 

1)  appropriate  CT  scans  before  and  after  injecting  the  contrast  agent;  and 

2)  appropriate  registration  and  subtraction  methods  between  the  two  scans 

may  significantly  improve  the  detection  of  aneurysms  and  their  quantitative  analysis  such 
as  their  location,  size  and  shape.  An  FDA  approved  workstation  (TeraRecon,  San  Mateo, 
California,  USA)  was  used  for  volumetric  rendering  as  shown  in  Figure  28(a).  The  result 
was  then  compared  to  traditional  digital  subtraction  angiography  (DSA),  the  current  gold 
standard,  and  3D  rotational  DSA  (rDSA)  for  image  correlation  (Figure  28(b)).  The 
aneurysm  morphology  such  as  overall  size,  neck  origin,  and  neck  size  is  better  seen  when 
a  skull  base  subtraction  technique  is  applied  to  intracranial  CTA.  These  important 
parameters  show  similar  image  correlation  on  skull  base  subtraction  CTA  images  as  they 
do  on  the  3D  rotational  DSA  images. 
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Figure  28:  (a)  Bone  subtraction  angiography  and 
(b)  rotational  digital  subtraction  angiography 


From  these  studies,  we  found  that  traditional  CTA  is  limited  at  the  level  of  skull  base 
where  osseous  structures  are  the  predominant  factor.  The  skull  base  subtraction  technique 
enhances  visualization  of  vascular  pathology  within  this  level.  This  subtraction  technique 
is  operator  non-dependent,  and  requires  little  additional  post-processing  effort.  Most 
importantly,  it  adds  additional  diagnostic  certainty  to  our  noninvasive  CTA 
armamentarium.  We  have  submitted  this  result  to  the  ASNR  (American  Society  of 
NeuroRadiology)  2005  annual  conference  [Choi  2005]. 


4  Key  Research  Outcomes 

This  section  provides  a  bulleted  list  of  key  research  accomplishments  during  the  entire 
project: 

•  Received  FDA  approval  to  conduct  a  clinical  trial  for  robotically  assisted  spinal 
blocks  and  completed  twenty  patients 

•  Completed  a  feasibility  demonstration  of  robotically  assisted  lung  biopsy  using 
frame-grabbed  CT  fluoroscopy  images  and  a  respiratory  motion  phantom 

•  Completed  an  initial  set  of  approved  animal  studies  demonstrating  the  feasibility 
of  electromagnetic  position  sensing  for  image  guidance  in  the  interventional  suite 

•  Gathered  skin  motion  data  on  four  patients  during  CyberKnife  stereotactic 
radiosurgery  treatments  under  an  IRB-approved  protocol 

•  Developed  a  respiratory  motion  simulator  consisting  of  two  3  degree  of  freedom 
platforms  to  evaluate  CyberKnife  treatments  which  compensate  for  internal  organ 
motion 
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5  Reportable  Outcomes 

This  section  provides  a  list  of  reportable  outcomes. 

The  major  product  of  this  year  is  the  list  of  papers  given  in  Section  9,  References.  Copies 
of  these  documents  are  provided  in  the  appendix. 

In  addition,  several  proposals  to  the  National  Institutes  of  Health  were  submitted  based 
on  this  work.  A  grant  was  received  from  NIH/NCI  to  develop  robotically  assisted  lung 
biopsy.  Several  small  business  grants  were  received  with  company  partners,  including 
one  on  electromagnetic  tracking  and  one  on  open  source  software. 

A  graduate  student  from  Catholic  University  and  a  graduate  student  from  Johns  Hopkins 
University  were  supported  to  assist  in  software  development  for  the  robotic  needle  driver. 
The  research  group  at  Georgetown  continued  to  take  a  lead  in  the  Washington  Area 
Computer  Aided  Surgery  Society  (www.washcas.org'),  which  was  formed  in  2000  to 
promote  research  in  the  field. 


6  Conclusions 

The  Periscopic  Spine  Surgery  project  has  continued  to  lay  the  ground  work  for 
developing  the  physician  assist  systems  of  the  future.  These  systems  will  incorporate 
robotics,  tracking,  and  visualization  to  improve  the  precision  of  instrument  placement  and 
manipulation  in  minimally  invasive  procedures.  The  world’s  first  clinical  trial  of 
robotically  assisted  needle  placement  for  spinal  nerve  blocks  was  begun  and  twenty 
patients  were  enrolled.  The  feasibility  of  electromagnetic  position  sensing  and  image 
overlay  was  demonstrated  in  an  animal  study.  A  respiratory  motion  simulator  was 
developed  to  evaluate  CyberKnife  treatments  which  compensate  for  internal  organ 
motion.  This  project  has  enabled  the  Georgetown  team  to  become  a  world  leader  in  the 
emerging  fields  of  computer  aided  surgery  and  medical  robotics.  Our  goal  will  continue 
to  be  to  develop  systems  to  add  the  physician  in  these  demanding  minimally  invasive 
procedures  with  the  ultimate  aim  of  improving  patient  care. 
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Abstract 

Force-reflecting  master  arms  are  explored  for  use 
as  haptic  displays  in  physical  therapy  interventions 
over  the  internet.  Rehabilitation  tasks  can  be  con¬ 
structed  in  which  both  the  patient  and  therapist 
can  interact  with  a  common  object  from  distant 
locations.  Each  haptic  master  exerts  “forces”  on  a 
virtual  object  which,  in  response,  generates  desired 
velocities  for  the  master  arm  to  track.  A  novel  coop¬ 
erative  control  architecture  based  on  wave  variables 
is  implemented  to  counter  the  destabilizing  effect  of 
internet  time-delay.  The  control  scheme  is  validated 
experimentally  using  a  pair  of  InMotion2  robots  in 
a  virtual  beam  manipulation  task  between  remote  sites. 


1  Introduction 

The  Imaging  Science  and  Information  Systems  (ISIS) 
Center  at  Georgetown  University  is  currently  engaged 
in  an  effort  to  develop  robotics  for  tele-rehabilitation. 
One  of  the  chief  goals  of  this  project  is  to  connect  phys¬ 
iotherapy  robots  in  remote  locations  for  clinical  assess¬ 
ment  and  treatment  of  patients  with  neuromuscular 
disorders.  A  pair  of  InMotion2  (IM2)  robots  from  In¬ 
teractive  Motion  Technologies  has  recently  been  inte¬ 
grated  into  a  testbed  at  the  ISIS  Center  to  facilitate 
this  effort  [10], 

The  IM2  “robot”  is  a  direct-drive,  four-bar  linkage 
with  a  planar  workspace  of  90  x  60  cm  and  maximum 
continuous  force  output  of  30  N  in  each  direction  (see 
Figure  1).  The  handle  is  pinned  to  the  distal  end  of  the 
outboard  link  providing  a  third,  unactuated  degree  of 
freedom.  The  linkages  result  in  an  apparent  stiffness 
of  6000  N/m  and  inertia  of  1.33  kg  at  the  handle. 
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The  low  inherent  dynamics  of  the  IM2  robot  makes 
it  ideally  suited  for  incorporating  haptic  virtual  real¬ 
ity  (VR)  into  our  strategy  for  physical  therapy  over 
the  internet.  This  will  allow  the  clinician  to  remotely 
assess  the  patient’s  condition  as  well  as  assist  the 
patient  perform  rehabilitation  tasks.  Implementing 
force-feedback  in  the  presence  of  time-delay  will  be 
a  key  factor  in  realizing  this  technology  over  the  in¬ 
ternet  and  is  a  cornerstone  of  the  control  architecture 
presented  here. 


Figure  1:  Four-bar  linkage  InMotion2  Robot  with 
graphic  display  of  virtual  beam  task. 


This  article  begins  with  a  brief  review  of  previous 
work  on  cooperative  haptics  and  time-delay  compen¬ 
sation  in  Section  2.  A  cooperative  beam  handling 
task  using  admittance  control  for  low-order  (process¬ 
ing)  time  delays  is  considered  in  Section  3.  A  con¬ 
trol  architecture  based  on  wave  variables  is  introduced 
in  Section  4  for  handling  larger  (internet)  time-delays. 
Experimental  results  on  the  IM2  testbed  for  both  types 
of  controllers  are  presented  in  Section  5  for  a  virtual 
beam  manipulation  task.  Conclusions  on  the  imple¬ 
mentation  of  this  approach  are  given  in  Section  6  along 
with  some  directions  for  future  research. 
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2  Previous  Work 

Simultaneous  application  of  multiple  haptic  devices 
has  been  attempted  on  several  VR  platforms.  A  pair 
of  2-DOF  master  manipulators  was  used  to  simulate 
thumb  and  index  fingertip  contact  with  an  object  dur¬ 
ing  a  peg- in- hole  insertion  task  [8,  2].  Dual-arm  con¬ 
tact  with  a  steering  wheel  was  simulated  using  a  pair 
of  6-DOF  PHANToM  devices  for  arm  motor  control 
training  [6].  NASA  researchers  are  using  a  pair  of 
Charlotte  IVA  robots  to  simulate  large  satellites  dur¬ 
ing  astronaut  training  [15,  3].  A  pair  of  6-DOF,  paral¬ 
lel  force  displays  was  used  at  the  University  of  Tsukuba 
for  performing  interactive  patient-therapist  tasks  over 
the  internet  [16].  Predictive  displays  were  used  to  help 
operators  adjust  for  up  to  3  sec  delays,  but  time-delay 
compensation  was  not  implemented. 

Several  investigators  have  incorporated  explicit 
time-delay  compensation  in  the  force-feedback  loops 
of  haptic  systems.  NASDA  used  a  straightforward 
approach  whereby  damping  was  increased  in  the  PD 
control  loop  to  maintain  stability  during  ground-based 
teleoperation  of  the  Japanese  ETS-VII  experiment  in 
1997  [9].  Scattering  theory  was  explored  by  [11]  to 
produce  passive  communications  during  teleoperation 
of  a  metal  block.  Wave  variables  were  introduced  by 
[14]  for  a  variety  of  master/slave  scenarios  with  widely 
varying  time  delay.  More  recently,  [1]  considered  time- 
delay  in  their  passivity  control  formulation  of  stable 
interaction  with  virtual  environments.  However,  none 
of  the  investigations  cited  considered  time-delay  com¬ 
pensation  in  the  context  of  multiple  haptic  displays. 
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Figure  2:  System  block  diagram  showing  haptic  master 
impedance  controller  and  virtual  object  generator. 


Figure  3:  Admittance  controller  block  diagram. 


3  Admittance  Control  for  Low 
Time-Delay 

The  system  block  diagram  for  a  cooperative  virtual 
object  manipulation  task  is  shown  in  Figure  2.  Each 
haptic  master  is  outputting  a  set  of  desired  forces  to  a 
virtual  object  generator  (VOG)  via  their  force/torque 
sensor  readings  and  in  response  receiving  a  set  of  de¬ 
sired  velocity  commands  to  track  from  the  virtual  ob¬ 
ject.  For  processing  delays  under  approximately  10 
msec,  this  strategy  produces  acceptable  behavior.  But 
when  the  delays  are  on  the  order  of  100  msec,  insta- 
bilility  is  easily  induced  in  the  system. 

The  control  block  diagram  for  the  master  arm  is 
shown  in  Figure  3.  This  is  a  classical  admittance  con¬ 
trol  implementation  in  which  the  haptic  master  is  us¬ 
ing  force  input  to  generate  a  set  of  desired  velocity 
commands  ([7], [4]).  The  velocity  commands  are  then 
tracked  using  a  PD  controller  implemented  in  Carte¬ 
sian  space  using  the  control  law  given  by 

Fc  =  J'm)  4"  ^m)  (I) 

where  Bm  and  Km  are  diagonal  damping  and  stiffness 
gains,  respectively,  and  Fc  is  the  commanded  force  rep¬ 
resented  in  Cartesian  “operational”  space. 


The  master  arm  dynamics  in  operational  space  axe 
computed  from 

Mx%m  +  be (xm ,  Xrn)  4“  Qx{x  m)  —  Fc  +  Fh  (2) 

where  Fh  is  the  force  applied  by  the  human  operator 
at  the  handle  [5].  For  haptic  devices  in  general,  the  in¬ 
ertial  Mx  and  Coriolis/centripetal  forces  cx  on  the  left 
side  of  (2)  are  low.  In  addition,  the  gravitational  force 
gx  for  the  IM2  robot  is  zero  since  the  planar  workspace 
is  perpendicular  to  the  gravity  vector.  Therefore,  the 
force  applied  by  the  human  Fh  is  approximately  equal 
but  opposite  the  commanded  force  Fc. 

The  VOG  calculates  the  dynamics  of  the  virtual  ob¬ 
ject  being  manipulated  by  the  master  arm.  In  the 
cooperative  beam  manipulation  task  in  Figure  4,  each 
haptic  master  is  grasping  one  end  of  a  beam  and  lift¬ 
ing  it  against  gravity.  The  grasp  point  is  the  handle 
pinned  at  the  distal  end  of  the  last  link,  and  the  simu¬ 
lated  gravity  vector  is  pointed  in  the  sagittal  plane  of 
the  operator  so  that  s/he  is  pushing  away  when  lifting 
the  beam  (toward  the  screen  in  Figure  1). 

The  position  of  the  left  end  of  the  beam  is  (a:;,, ,  yi,L ) 
and  the  right  side  is  (xb2,yb2)-  The  orientation  of  the 
beam  with  respect  to  the  a:o-axis  is  given  by  8b-  The 
total  length  of  the  beam  is  2 L  so  that  the  relationship 
between  the  left  and  right  position  is  given  by 
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Figure  4:  Cooperative  beam  manipulation  task. 


“wave”  command  is  issued  by  the  master,  and  then 
an  impedance  “wave”  is  reflected  by  the  slave.  How 
much  of  the  push  wave  is  reflected  depends  upon  the 
impedance  on  the  slave  side.  A  soft,  spring-like  envi¬ 
ronment  has  a  low  impedance  so  will  not  reflect  the 
incoming  wave  as  greatly  as  a  rigid  environment.  In 
addition  to  its  impedance/admittance  analog,  the  per¬ 
formance  of  the  wave  variable  controller  degrades  only 
gradually  with  increasing  time  delay  and  so  is  well- 
suited  to  the  environment  of  the  internet. 

When  using  the  wave  variable  approach,  it  is  im¬ 
portant  to  distinguish  between  which  object(s)  is  the 
“master”  and  which  is  the  “slave” .  In  our  application, 
the  virtual  object  is  regarded  as  the  “slave”  and  is 
acted  upon  by  virtual  forces  from  the  two  IM2  robots 
which  are  the  “masters”.  Each  of  these  “applied” 
forces  are  computed  from  the  transmitted  wave  vari¬ 
able  from  the  master  as  follows 


xb2  =  xbl  +  2  Lcos(9b)  (3) 

Vb2  =  2/f>i  +  2  Lsin(9b) 

The  center  of  mass  of  the  beam  (CM)  is  chosen  to  be 
at  the  geometric  center,  and  the  beam  is  assumed  to 
be  a  uniform  slender  rod  so  that  the  inertia  about  its 
center  of  mass  is  given  by  ib  =  | mbL 2. 

The  resulting  beam  dynamics  are  given  by 


Mf,(xb)xb  +  Cb(xb,  ib )  =  Cp1Fi  -(-  Cf2F 2  +  ixib^g  (4) 


where 
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where  cb  =  cos(6b),  sb  =  sin(db),  and  aj  =  [0  —  g  0] 
is  the  gravitational  acceleration  vector. 


4  Wave  Variable  Control 

There  are  many  approaches  for  dealing  with  instability 
caused  by  time  delay,  but  the  wave  variable  approach 
seems  to  produce  a  natural  interface  for  manipulating 
objects  over  the  internet  [13,  14].  Instead  of  send¬ 
ing  a  direct  velocity  or  force  command,  an  impedance 


Fs  =  —  bxs  +  \/2 bus  (5) 


The  VOG  then  computes  its  outgoing  wave  variable 
using 


_  bx§  Fs 

~  y/2b 


(6) 


The  incoming  wave  to  the  master  is  then 
Vm(t)  =  Vm(t-T). 

The  desired  master  velocity,  xmd>  is  computed  from 
the  master  force  and  return  wave  variable  vm  as  fol¬ 
lows: 


_  bxmd  +  Fm 

V2b 


(7) 


The  incoming  wave  to  the  slave  is  then  given  by 
us(t)  —  um(t  —  T).  If  the  master  force  command  Fc 
is  used  to  compute  the  master  force  in  (7),  Fm  =  —Fc 
and  (1)  and  (7)  form  a  recursive  loop.  Substituting 
(1)  into  (7)  and  solving  for  xmd  gives 

V*2 ~bVm  T  JBmXjn  T 


md 


•E md 


=  /  Xmd{r)d7 

Jo 


Bm  +  b 


(8) 


The  control  architecture  for  the  cooperative  task  us¬ 
ing  a  wave  variable  implementation  is  shown  in  Fig¬ 
ure  5.  In  this  scenario,  each  haptic  master  is  inde¬ 
pendently  interacting  with  the  virtual  object  which 
is  considered  the  “slave”.  The  virtual  object  process 
then  decodes  the  wave  variables  from  the  masters  into 
a  set  of  force  commands  to  be  imparted  on  the  slave. 
The  virtual  object  generator  calculates  the  resultant 
motion  at  each  of  the  “contact”  points  and  transmits 
it  via  wave  variable  back  to  the  master.  Each  master 
control  process  then  decodes  the  slave  wave  variables 
into  desired  velocities  at  its  “contact”  point.  The  de¬ 
sired  velocity  vector  (along  with  the  integrated  value) 
is  then  input  to  the  master’s  PD  tracking  controller. 
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Figure  5:  Wave  variable  control  architecture. 

The  virtual  object  dynamics  are  calculated  via  a  sep¬ 
arate  process  on  one  of  the  master  arm  computers.  T\ 
and  T2  are  time  delays  caused  by  either  internet  tran¬ 
sit  or  computational  processing.  If  the  beam  dynamics 
are  being  calculated  on  the  master  1  computer,  then 
Tj  is  primarily  the  computational  delay  for  the  beam 
dynamics  process  (essentially  zero),  and  T 2  is  the  in¬ 
ternet  time  delay  for  a  signal  to  reach  master  1  from 
master  2.  However,  to  maintain  a  truly  cooperative 
task,  the  two  time  delays  should  be  matched.  There¬ 
fore,  an  artificial  time  delay  based  on  a  moving  average 
of  the  internet  time  delay  is  applied  to  the  master  con¬ 
trol  computer  hosting  the  virtual  object  process. 

The  effect  of  an  increase  in  time  delay  on  the  wave 
variable  implementation  is  to  decrease  the  system’s 
natural  frequency.  This  communications  “stiffness” 
Kcomm  is  given  by  b/T.  Thus  b  should  be  increased 
in  proportion  to  the  time  delay  to  maintain  the  stiff¬ 
ness  in  the  system  [12].  For  a  processing  time  delay 
of  0.075  msec,  6=5  was  found  to  produce  good  track¬ 
ing  performance.  For  roundtrip  time  delays  of  50-100 
msec,  6=40  was  required  to  maintain  similar  perfor¬ 
mance. 

5  Dual-Arm  Beam  Manipula¬ 
tion  Experiment 

The  architecture  of  the  IM2  rehabilitation  testbed  is 
shown  in  Figure  6.  Each  operator  receives  both  vi¬ 
sual  (GUI)  and  haptic  input  (IM2)  from  the  system 
which  is  being  driven  by  an  AMD  XP1800  PC  with 
Athlon  686  processor  running  at  1533  MHz.  The  con¬ 
trol  process  was  implemented  in  RT-Linux  and  uses 
approximately  1.2%  of  the  CPU  time  at  a  rate  of  200 
Hz.  Fast  internet  communication  between  robots  was 
achieved  using  UDP  protocol  which  enabled  transfer 
rates  of  100  Hz  for  typical  tests.  The  time  delay  for 
the  therapist’s  computer,  T\,  was  set  equal  to  the  in¬ 
ternet  time  delay  T  to  maintain  symmetry  in  the  force 


interaction  between  the  VOG  and  two  robots. 

Two  sets  of  experiments  were  performed  to  illustrate 
the  cooperative  beam  manipulation  task  over  the  in¬ 
ternet:  admittance  control  under  just  processing  delay, 
and  wave  variable  control  in  the  presence  of  internet 
time  delay.  In  all  tests,  the  master  controller  had  a 
bandwidth  of  30  rad/sec  and  was  critically  damped 
yielding  gains  of  ATm=900  N/m  and  Bm=  60  N/m/s. 
The  beam  parameters  were  rrib= 10  kg,  L=0.15  m,  and 
4=0.075  kg-m2.  A  reduced  gravitational  acceleration 
of  g= 3  m/s2  was  used  in  order  not  to  exceed  the  force 
capacity  of  the  robot.  Parameters  could  be  changed  by 
the  operator  using  the  graphical  user  interface  shown 
previously  in  Figure  1. 


Figure  6:  Hardware  configuration  of  the  system. 


In  the  first  set  of  tests,  the  impedance  control 
scheme  of  Figure  2  was  used.  The  control  and  commu¬ 
nication  rates  were  200  Hz,  and  the  time  delay  within 
our  own  IP  domain  was  only  0.15  msec.  The  beam 
starts  out  horizontally  and  then  is  lifted  by  the  haptic 
master  on  the  left  until  it  reaches  the  vertical  position. 
Then  the  second  haptic  master  raises  the  right  side  of 
the  beam  until  it  is  again  horizontal.  The  plot  of  the 
beam  angle  9b  versus  time  is  shown  in  Figure  7. 

The  plots  of  the  commanded  vertical  forces  on  the 
beam  (sensed  master  forces)  are  shown  in  Figure  8. 
Fy  for  haptic  master  1  is  seen  to  go  to  zero  when  the 
beam  reaches  a  vertical  position  while  haptic  master 
2  sustains  the  full  load  of  the  beam.  After  master  2 
raises  its  side  of  the  beam,  the  force  becomes  equally 
distributed  again.  The  desired  versus  actual  velocities 
are  shown  in  Figure  9  for  master  1  and  indicate  very 
good  tracking  by  the  PD  controller. 

In  the  second  set  of  tests,  the  wave  variable  control 
scheme  of  Figure  5  was  used.  The  controller  rate  was 
decreased  to  100  Hz  due  to  the  bandwidth  limitation  of 
the  communication  process,  and  the  wave  impedance 
parameter  6  was  increased  to  40  to  compensate  for  the 
additional  delay.  An  artificial  communication  time  de¬ 
lay  of  approximately  50  msec  was  introduced  between 
the  therapist’s  commands  and  the  virtual  object  to 
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Figure  7:  Beam  angle  under  admittance  control.  Figure  10:  Beam  angle  under  wave  variable  control. 
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Figure  8:  Y-contact  forces  under  admittance  control. 
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Figure  9:  Desired  and  actual  velocity  of  master  1  under 
admittance  control. 
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Figure  11:  Y-contact  forces  under  wave  variable  con¬ 
trol. 


Figure  12:  Desired  and  actual  velocity  of  master  1 
under  wave  variable  control. 


balance  the  internet  communication  delay  between  the 
two  computers. 

The  beam  was  manipulated  in  the  same  manner 
as  before  yielding  the  beam  angle  6b  shown  in  Fig¬ 
ure  10.  The  commanded  master  and  slave  forces  in 
the  y— direction  for  the  two  haptic  masters  are  shown 
in  Figure  11.  Fy  for  both  masters  starts  out  equal  and 
then  goes  to  zero  for  master  1  when  the  beam  reaches 
a  vertical  position  and  master  2  sustains  the  full  load 
of  the  beam.  The  desired  versus  actual  velocities  for 
haptic  master  1  shown  in  Figure  12  indicate  very  good 
tracking  by  the  wave  controller. 

The  roundtrip  internet  time  delay  between  George¬ 
town  University  and  Cambridge,  Mass,  for  this  test  is 
shown  in  Figure  13.  The  time  delay  varied  between  35 
and  110  msec  and  averaged  about  50  msec.  A  10  sec 


window  was  used  to  compute  a  moving  average  for  the 
artificial  delay  Xi  to  be  applied  to  master  1. 


6  Conclusion 

This  paper  has  outlined  a  strategy  for  cooperative, 
force-feedback  control  of  virtual  objects  over  the  in¬ 
ternet  based  on  using  impedance  control  for  low  time- 
delay  and  wave  variables  for  large  time  delay.  Experi¬ 
ments  indicate  that  the  impedance  control  approach  is 
stable  for  round  trip  time  delays  under  about  90  msec 
but  degrades  rapidly  thereafter.  Above  100  msec  de¬ 
lay,  the  robot  handle  must  be  grasped  firmly  or  any 
imparted  forces  produce  rapidly  divergent  behavior. 
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Figure  13:  Internet  roundtrip  time  delay. 

By  contrast,  the  wave  variable  approach  was  stable 
for  a  wide  range  of  time  delays  tested,  including  arti¬ 
ficial  delays  of  over  1  sec.  The  commanded  force  must 
be  used  to  generate  the  wave  variables  because  the 
sensed  force  (which  neglects  the  haptic  dynamics)  vi¬ 
olates  passivity.  Very  good  tracking  of  the  desired  dy¬ 
namics  was  achieved  in  an  actual  task  performed  over 
the  internet  for  time  delays  varying  between  35  and  110 
msec.  However,  the  time-delay  compensation  comes  at 
the  cost  of  perceived  heavier  inertia  and  damping  at 
the  handle  due  to  the  increased  wave  impedance. 

Future  work  will  focus  on  generating  additional  co¬ 
operative  tasks  and  experimenting  with  longer  time 
delays  that  are  more  characteristic  of  broadband  sys¬ 
tems  (T1  lines  used  in  these  tests).  In  addition,  the 
wave  impedance  approach  will  be  modified  to  account 
for  the  variable  time  delay  [14],  and  methods  of  coun¬ 
tering  the  effects  of  increased  wave  impedance  will  be 
investigated.  Use  of  high  bandwidth  force  sensing  is 
currently  being  incorporated  into  the  controller  to  in¬ 
crease  the  fidelity  of  the  haptic  interface. 
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ABSTRACT 

The  feasibility  of  performing  remote  assessment  and  therapy  of  patients  over  the  internet  using 
robotic  devices  is  explored.  Using  a  force  feedback  device,  the  therapist  can  assess  the  range 
of  motion,  flexibility,  strength,  and  spasticity  of  the  patient’s  arm  grasping  a  similar  robotic 
device  at  a  remote  location.  In  addition,  cooperative  rehabilitation  strategies  can  be  developed 
whereby  both  the  patient  and  therapist  cooperatively  perform  tasks  in  a  virtual  environment. 
To  counter  the  destabilizing  effects  of  time  delay  in  the  force  feedback  loop,  a  passive  wave 
variable  architecture  is  used  to  encode  velocity  and  force  information.  The  control  scheme  is 
validated  experimentally  over  the  internet  using  a  pair  of  InMotion2  robots  located  500  miles 
apart. 


1.  INTRODUCTION 

Robots  have  been  explored  as  possible  rehabilitation  aids  in  laboratory  settings  for  well  over  a  decade.  These 
investigations  have  recently  expanded  into  the  field  of  “teletherapy”  whereby  a  clinician  can  interact  with 
patients  in  remote  locations  using  robotic  devices.  However,  time  delays  encountered  in  the  force  feedback 
loop  can  cause  instability  in  the  system.  Compensating  for  the  time  delay  will  be  key  to  realizing  this 
technology  over  the  internet  and  is  a  cornerstone  of  the  control  architecture  presented  here. 

The  specific  aims  of  our  research  are  twofold:  to  enable  a  clinician  to  assess  the  physical  condition  of  a 
patient’s  arm  using  metrics  such  as  strength,  dexterity,  range  of  motion,  and  spasticity;  and  to  help  a  clinician 
perform  cooperative  rehabilitative  tasks  with  a  patient  using  a  virtual  environment  intended  to  simulate  active 
daily  living  (ADL)  tasks.  The  use  of  a  “haptic”  (force-feedback)  device  in  conjunction  with  a  video  display 
will  allow  the  clinician  to  remotely  assess  the  patient’s  condition  as  well  as  assist  the  patient  while 
performing  rehabilitation  tasks.  The  ISIS  Center  at  Georgetown  University  Medical  Center  has  recently 
assembled  a  robot  rehabilitation  test  bed  consisting  of  a  pair  of  InMotion2  (IM2)  robots  from  Interactive 
Motion  Technology,  Inc.  (Krebs  et  al,  2001).  The  IM2  Robot  is  a  direct-drive,  four-bar  linkage  with  a  planar 
workspace  of  90  x  60  cm  and  maximum  continuous  force  output  of  30  N  in  each  direction  (see  Figure  1). 
The  handle  is  pinned  to  the  distal  end  of  the  outboard  link  providing  a  third,  unactuated  degree  of  freedom. 
The  apparent  mass  at  the  handle  is  only  1 .33  kg  making  it  well-suited  to  our  dual  purposes. 

This  article  begins  with  a  brief  review  of  previous  work  on  internet  therapy  and  cooperative  haptics  in 
Section  2.  The  tele-assessment  and  cooperative  rehabilitation  modes  are  described  in  Section  3.  The  haptic 
controller  and  time-delay  compensation  using  wave  variables  are  outlined  in  Section  4.  Experimental  results 
for  both  operational  modes  implemented  on  the  IM2  test  bed  are  presented  in  Section  5.  Conclusions  and 
future  research  are  discussed  in  Section  6. 


2.  PREVIOUS  WORK 

Telemedicine  has  already  seen  several  successful  demonstrations  of  rehabilitation  robotics.  A  “java  therapy” 
application  was  enabled  using  a  commercial,  force-feedback  joystick  connected  to  an  orthopaedic  splint 
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(Reinkensmeyer,  2001).  Patients  log  into  the  website  “javatherapy.com”  and  a  physical  or  occupational 
therapist  will  guide  them  through  a  repetitive  movement  regimen  intended  to  improve  their  sensorimotor 
skills.  Such  therapy  has  been  demonstrated  to  be  useful  even  several  years  following  hemiplegic  stroke. 

The  Rutgers  Haptic  Master  11  (RMII)  was  used  to  increase  hand  strength  in  stroke  patients  using 
teletherapy  (Popescu,  2000).  When  the  patient  picks  up  an  object  such  as  the  chess  piece  seen  on  a  computer 
screen  in  Figure  2,  the  computer  actuates  the  piezoelectric  servo  valves  on  the  hand  exoskeleton  to  provide 
resistive  “grasp”  forces  to  the  hand.  The  remote  therapist  can  modify  this  resistance  during  the  sessions  to 
increase  the  patient’s  strength  and  also  design  an  increasing  complex  array  of  virtual  tasks  for  the  patient  to 
perform  to  flirther  challenge  their  motor  skills.  Pilot  clinical  trials  on  post-stroke  patients  have  indicated 
hand  mechanical  work  increase  using  the  RMII  (Burdea,  2001). 


Figure  1.  InMotion2  Robot  with  graphic  display  of 
virtual  beam  task. 


Cooperative  control  using  haptic  devices  has  been  attempted  on  several  virtual  reality  platforms.  A  pair  of  2- 
DOF  master  manipulators  was  used  to  simulate  thumb  and  index  fingertip  contact  with  an  object  during  a 
peg-in-hole  insertion  task  (Howe  and  Kontarinis,  1992;  Burdea,  1996).  Dual-arm  contact  with  a  steering 
wheel  was  simulated  using  a  pair  of  6-DOF  PHANToM  devices  for  arm  motor  control  training  (Goncharenko 
et  al,  2003).  Yano  and  Iwata  (1995)  used  a  pair  of  6-DOF,  parallel  mechanism  force  displays  to  perform 
interactive  patient-therapist  tasks  over  the  internet.  Although  predictive  displays  were  used  to  help  operators 
adjust  for  up  to  3  sec  delays,  explicit  time-delay  compensation  was  not  implemented. 

Several  investigators  have  incorporated  explicit  time-delay  compensation  in  the  force-feedback  loops  of 
haptic  systems.  Scattering  theory  was  explored  by  Lawn  and  Hannaford  (1993)  to  produce  passive 
communications  during  teleoperation  of  a  metal  block.  Wave  variables  were  introduced  by  Niemeyer  and 
Slotine  (1997)  for  a  variety  of  master/slave  scenarios  with  widely  varying  time  delay.  Adams  and  Hannaford 
(1999)  also  considered  time-delay  in  their  passivity  control  formulation  of  stable  interaction  with  virtual 
environments.  However,  none  of  the  investigations  we  encountered  considered  time-delay  compensation  in 
the  context  of  multiple  haptic  displays. 


Figure  2.  Rutgers  Haptic  Master  being  used  to  reflect 
grasp  force  during  virtual  chess  match. 


3.  OPERATING  MODES 

The  robot  test  bed  has  two  operating  modes:  Tele-Assessment  and  Cooperative  Rehabilitation.  In  Tele- 
Assessment  Mode,  the  clinician  attempts  to  evaluate  various  properties  of  the  patient’s  arm  through  bilateral 
manipulation  over  the  internet.  In  Cooperative  Rehabilitation  Mode,  the  patient  and  therapist  cooperatively 
manipulate  common  objects  over  the  internet  by  moving  their  robot  handles  to  accomplish  a  therapeutic  task. 
Both  modes  are  described  in  detail  below. 

3. 1  Tele-Assessment  Mode 

In  this  mode,  the  robot  handle  that  is  being  grasped  by  the  subject  mirrors  movements  made  by  the  clinician’s 
robot  and  vice  versa.  A  force  sensor  on  the  patient’s  robot  relays  forces  exerted  by  the  subject  back  to  the 
clinician’s  robot  where  the  force  pattern  will  be  “displayed”  on  the  haptic  interface.  This  position-based 
“force-reflection”  is  commonly  used  today  in  robot-assisted  surgery. 

The  system  block  diagram  for  assessment  mode  is  shown  in  Figure  3  and  is  similar  to  the  bilateral  force 
feedback  architecture  used  in  master/slave  teleoperation.  Both  the  master  and  slave  are  under  Cartesian  PD 
control  where  the  position  of  the  master  becomes  the  desired  position  of  the  slave,  and  the  position  of  the 
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slave  becomes  the  desired  position  of  the  master  (same  holds  for  velocity).  The  position  and  velocity  data  for 
each  robot  is  “packetized”,  sent  across  the  internet  using  an  internet  socket,  and  picked  up  by  a 
communication  process  at  the  other  side  where  it  is  unpacked  and  used  by  the  local  controller. 

Since  the  PD  controller  filters  out  high  frequency  content  from  the  patient’s  arm  that  might  be  useful  for 
patient  assessment,  a  force-sensor  capable  of  picking  up  high  frequency  phenomena  such  as  hand  tremor  was 
used  to  augment  the  haptic  display.  The  force  sensor  output  is  high-pass  filtered  and  transferred  alongside 
the  position/velocity  data  to  the  therapist’s  robot  where  it  is  amplified  by  a  gain  k  and  added  to  the  PD 
control  input.  The  high  pass  filter  is  necessary  to  remove  bias  readings  normally  present  in  the  force  sensor 
that  would  otherwise  cause  a  position  offset  (Murphy,  1994). 


Figure  3.  Bilateral  Tele-Assessment  Mode  Architecture. 

3.2  Cooperative  Rehabilitation  Mode 

The  control  architecture  for  the  cooperative  task  is  shown  in  Figure  4.  In  this  scenario,  both  the  therapist  and 
patient  robots  are  considered  “masters”  which  are  independently  interacting  with  the  virtual  object  which  is 
considered  the  “slave”.  The  virtual  object  generator  (VOG)  applies  the  sensed  “interaction”  forces  from  the 
masters  and  then  calculates  the  resultant  motion  of  the  object.  The  motion  of  the  object  at  each  “contact” 
point  is  then  transmitted  back  to  each  master  where  it  is  tracked  by  a  controller. 


ARTIFICIAL 


DELAY 

Comm 
Process 
(IP  Socket) 

■nrf 

■Hj_> 

i 

f 

i 

VIRTUAL  OBJECT 


PC-686 

1 

GRAPHICS 

- H 

CONTROL 

1 

1  f 

1 

Robot  #1 


Therapist 


Comm 
Process 
(IP  Socket) 

L 

PC-686 

-  GRAPHICS 

CONTROL 

1 

j 

r 

h 

Patient 


Robot  $2 


Figure  4.  Hardware  configuration  for  Cooperative  Rehabilitation  Mode. 

The  virtual  object  dynamics  are  calculated  via  a  separate  process  on  one  of  the  master  arm  computers.  Ti  and 
T2  are  time  delays  caused  by  either  internet  transit  or  computational  processing.  If  the  object  dynamics  are 
being  calculated  on  the  master  1  computer,  then  Ti  is  primarily  the  computational  delay  for  the  VOG  process 
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(essentially  zero),  and  T2  is  the  internet  time  delay  for  a  signal  to  reach  master  1  from  master  2.  However,  to 
maintain  a  truly  cooperative  task,  the  two  time  delays  should  be  matched.  Therefore,  an  artificial  time  delay 
based  on  a  moving  average  of  the  internet  time  delay  is  applied  to  the  master  control  computer  hosting  the 
virtual  object  process  (master  1  in  this  case). 


4.  HAPTIC  CONTROL  AND  TIME  DELAY  COMPENSATION 

In  both  operating  modes,  the  core  of  the  haptic  controller  is  a  Cartesian  PD  controller  that  servos  on  the 
position  and  velocity  of  the  handle 

Fc=Bm{xmll-xm)  +  Km(xmd-xm)  (1) 

where  xm  is  the  position  of  the  handle,  Fc  is  the  commanded  Cartesian  force,  and  Bm  and  Km  are  diagonal 
damping  and  stiffness  gains,  respectively.  For  the  cooperative  mode,  an  additional  force  loop  wraps  around 
the  servo  loop  as  shown  in  Figure  5  to  provide  compliance  (Carignan  and  Cleary,  2000).  For  the  cooperative 
mode  realization  shown  in  Figure  6,  the  “sensed”  human  forces  applied  at  each  handle  are  used  as  the  force 
inputs  to  the  virtual  object  dynamics  to  generate  the  motion  command  inputs  to  each  master. 


The  haptic  controller  works  well  for  the  interconnected  robot  configurations  shown  in  Figures  3  and  4  as  long 
as  the  roundtrip  time  delay  is  under  about  100  msec.  As  the  time  delay  starts  to  exceed  100  msec,  the 
passivity  of  the  controller  becomes  severely  compromised  and  can  drive  the  system  unstable.  To  restore 
passivity  in  the  system,  compensation  using  wave  variables  emerged  as  the  most  natural  approach  for 
performing  cooperative  tasks  over  the  internet  (Niemeyer  and  Slotine,  1998). 

The  wave  variable  architecture  for  the  cooperative  mode  is  shown  in  Figure  7.  The  strategy  is  similar  for 
tele-assessment  mode  except  that  the  second  master,  rather  than  the  virtual  object,  is  the  “slave”.  Instead  of 
using  the  sensed  force  to  impart  force  commands  to  the  slave,  force  and  velocity  data  are  used  by  the  master 
to  generate  an  impedance  “wave”  command  that  is  transmitted  and  decoded  by  the  slave  side  into  a  force 
command  for  cooperative  rehabilitation  mode  or  a  velocity  command  for  tele-assessment  mode.  Part  of  the 
incoming  wave  is  subsequently  reflected  back  to  the  master.  How  much  of  the  wave  is  reflected  depends 
upon  the  impedance  of  the  slave;  a  yielding  environment  will  not  reflect  the  incoming  wave  as  greatly  as  a 
rigid  wall.  The  wave  impedance  b  is  a  tuning  parameter  used  to  trade-off  speed  and  force;  a  high  b  produces 
an  inertially  dominant  system,  and  a  low  b  presents  a  more  rigid  interface  (Niemeyer  and  Slotine,  1997b). 

Each  force  to  be  “applied”  to  the  slave  is  computed  from  the  transmitted  wave  variable  from  the  master 
using 

F]  =  -bxs  +  y[2bus  (2) 


where  the  incoming  wave  to  the  slave  uft)  is  the  delayed  output  wave  from  the  master,  um(t-T).  After  the 
virtual  object  dynamics  are  computed,  the  virtual  object  generator  emits  its  outgoing  wave  variable  using 


vs  = 


■yf2b 


(3) 


where  the  incoming  wave  to  the  master  vm(t)  is  the  delayed  output  wave  from  the  slave,  vs(t-T). 

The  desired  master  velocity,  dxmd/dt,  is  computed  from  the  master  force  Fra  and  return  wave  variable  vra  as 
follows.  The  outgoing  wave  from  the  master  is 


(4) 
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bxmJ  +  Fm 

Jib 


If  the  master  force  command  Fc  is  used  to  compute  the  master  force  in  (4),  then  Fc=-Fm  and  Eq  (1)  and  Eq  (4) 
form  a  recursive  loop  (Niemeyer  and  Slotine,  1997).  Substituting  Eq  (1)  into  Eq  (4)  and  solving  for  dxmd/dt 
gives 


V  26v„,  +  BJ„  + 
B„,+b 


xmd  =  [xmd{r)dT  (5) 

Note  that  the  wave  impedance  for  both  masters  was  chosen  to  be  the  same  since  the  time  delays  were 
matched  and  the  devices  were  identical. 

The  effect  of  an  increase  in  time  delay  on  the  wave  variable  implementation  is  to  decrease  the  system’s 
natural  frequency.  The  “communications  stiffness”  Kcomm  is  given  by  b/Tdeiay,  thus  the  wave  impedance 
should  be  increased  in  proportion  to  the  time  delay  to  maintain  system  bandwidth  (Niemeyer  and  Slotine, 
1997).  However,  the  time  delay  also  introduces  an  apparent  mass  proportional  to  delay,  Mcomm=bTdeiay, 
which  produces  a  heavier  feel  at  the  handle  as  the  time  delay  (or  wave  impedance)  increases.  Thus,  a  trade¬ 
off  exists  in  wave  impedance  between  maintaining  high  system  bandwidth  and  low  inertia  at  the  handle. 
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Figure  7.  Wave  variable  control  architecture. 


5.  EXPERIMENTAL  RESULTS 

The  control  station  and  haptic  controller  operate  on  an  AMD  XP1800  PC  with  Athlon  686  processor  running 
at  1533  MFIz.  The  control  process  was  implemented  in  RT-Linux  and  uses  approximately  1.2%  of  the  CPU 
time  at  a  rate  of  200  Hz.  Fast  internet  communication  between  robots  was  achieved  using  UDP  protocol 
which  enabled  transfer  rates  of  100  Hz  for  16  byte  datasets.  A  3rd-order  Butterworth  filter  with  a  5  Hz  cut¬ 
off  was  used  for  the  high-pass  filter  in  the  assessment  tests  (Fisher,  1999).  The  time  delay  for  the  therapist’s 
computer,  Ti,  was  set  equal  to  the  internet  time  delay  T2  in  the  rehabilitation  tests  to  maintain  symmetry 
between  the  VOG  and  each  robot. 

5. 1  Tele-Assessment 

As  a  demonstration  of  the  utility  of  the  high  bandwidth  force  feedback  in  assessment  mode,  an  experiment 
was  conducted  in  which  an  operator  used  the  master  robot  to  move  the  slave  robot  along  the  vertical  edge  of 
a  spiral  bound  notebook  as  shown  in  Figure  8.  The  operator  tried  to  maintain  a  constant  normal  force  as  the 
handle  moved  along  the  edge.  Figure  9  shows  the  total  force  command  for  the  master  robot  in  the  x- 
direction,  Fc,  superposed  on  just  the  PD  control  input  force  for  a  force  gain  of  k=5.  The  ripple  in  Fc  was  due 
to  the  force  sensor  picking  up  the  “tremor”  caused  by  the  spiral  edge  which  was  totally  missed  by  the  PD 
controller  which  had  a  bandwidth  of  approximately  5  Hz. 
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Figure  8.  Detecting  the  rough  edge  of  a  spiral 
notebook  during  tele-assessment  test. 


.Figure  9.  Total  force  command  (Fc)  and  PD 
component  (FPO)  in  x-direction  for  gain  of  k=5. 


5.2  Cooperative  Rehabilitation 


An  example  of  a  cooperative  rehabilitation  task  is  depicted  in  Figure  10.  The  patient  and  therapist  “pick  up” 
opposite  ends  of  a  virtual  beam  by  grasping  the  handle  which  coincides  with  the  end  of  the  beam.  Object 
parameters  such  as  mass,  length,  and  inertia  can  be  adjusted  to  correspond  to  real-life  objects  using  a 
Graphical  User  Interface  (GUI)  on  the  therapist’s  computer.  The  gravity  vector  points  in  the  sagital  plane  of 
the  operator  so  that  s/he  is  pushing  away  when  lifting  the  beam  (toward  the  screen  in  Figure  1).  As  the  object 
is  “lifted”,  the  side  that  is  lower  will  begin  to  feel  more  of  the  weight  thus  stimulating  the  participants  to 
maintain  the  beam  in  a  horizontal  position.  Also,  if  one  side  tugs  on  the  object,  the  other  side  feels  it  thus 
encouraging  a  cooperative  strategy  to  lift  the  object. 

The  VOG  calculates  the  dynamics  of  the  virtual  object  being  manipulated  by  the  master  arms.  The  centre 
of  mass  of  the  beam  is  chosen  to  be  at  the  geometric  centre,  and  the  beam  is  assumed  to  be  a  uniform  slender 
rod  so  that  the  inertia  about  its  centre  of  mass  is  given  by  ib=mbL2/3.  The  orientation  of  the  beam  with  respect 
to  the  x0-axis  is  given  by  0b  and  the  total  length  of  the  beam  is  2L.  The  resulting  beam  dynamics  are  given 
by 


Mb{xh)xb  +  cb(xi,xb)  =  CFF]  +CF2F2  +  mbag  (6) 

where  the  gravitational  acceleration  vector  is  a8=[0  -g  0]T.  The  complete  dynamics  for  Eq  (6)  can  be  found 
in  (Carignan  and  Olsson,  2004). 


IMBTERI1  HI  ASTER*  2 


Figure  10.  Cooperative  beam  manipulation  task. 

Three  sets  of  experiments  were  performed  to  illustrate  the  cooperative  beam  manipulation  task  over  the 
internet:  admittance  control  with  negligible  delay,  wave  variable  control  for  an  actual  internet  test,  and  wave 
variable  control  for  simulated  internet  roundtrip  time  delays  of  0.5  and  1  sec.  In  all  tests,  the  master 
controller  had  a  bandwidth  of  30  rad/sec  and  was  critically  damped  yielding  gains  of  Km=900  N/m  and 
Bm=60  N/m/s.  The  beam  parameters  were  mb=10  kg,  L=0.15  m,  and  ib=0.075  kg-m2.  A  reduced  gravitational 
acceleration  of  g=3  m/s2  was  used  in  order  not  to  exceed  the  force  capacity  of  the  robot.  Parameters  could  be 
changed  by  the  operator  using  the  graphical  user  interface  shown  previously  in  Figure  1. 

In  the  first  set  of  tests,  the  robots  were  colocated  at  the  ISIS  Center  and  the  admittance  control  scheme  of 
Figure  6  was  used.  The  control  and  communication  rates  were  200  Hz,  and  the  time  delay  within  our  own  IP 
domain  was  only  0.15  msec.  The  beam  starts  out  horizontally  and  then  is  lifted  by  the  haptic  master  on  the 
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left  until  it  reaches  the  vertical  position.  Then  the  second  haptic  master  raises  the  right  side  of  the  beam  until 
it  is  again  horizontal.  The  plot  of  the  beam  angle  0b  versus  time  is  shown  in  Figure  1 1 . 

The  plots  of  the  commanded  vertical  forces  on  the  beam  (sensed  master  forces)  are  shown  in  Figure  12. 
Fy  for  haptic  master  1  is  seen  to  go  to  zero  when  the  beam  reaches  a  vertical  position  while  haptic  master  2 
sustains  the  full  load  of  the  beam.  After  master  2  raises  its  side  of  the  beam,  the  force  becomes  equally 
distributed  again.  The  desired  versus  actual  velocities  for  master  1  (not  shown)  indicate  very  good  tracking 
by  the  PD  controller. 
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Figure  11.  Beam  angle  for  zero  time-delay  test.  Figure  12.  Vertical  force  applied  in  zero  delay  test. 

In  the  second  set  of  tests,  the  wave  variable  control  scheme  of  Figure  7  was  used.  The  controller  rate  was 
decreased  to  100  Hz  due  to  the  bandwidth  limitation  of  the  communication  process,  and  the  wave  impedance 
parameter  b  was  set  to  40  to  compensate  for  the  additional  delay.  The  roundtrip  internet  time  delay  between 
Georgetown  University  and  Cambridge,  Mass,  for  this  test  varied  between  35  and  1 10  msec  and  averaged 
about  50  msec.  A  10  sec  window  was  used  to  compute  a  moving  average  for  the  artificial  delay  Ti  to  be 
applied  to  master  1 . 


The  beam  was  manipulated  in  the  same  manner  as  before  yielding  the  beam  angle  0b  shown  in  Figure  13. 
The  commanded  master  and  slave  forces  in  the  y-direction  for  the  two  haptic  masters  are  shown  in  Figure  14 
and  look  remarkably  similar  to  the  zero-delay  test.  Fy  for  both  masters  starts  out  equal  and  then  goes  to  zero 
for  master  1  when  the  beam  reaches  a  vertical  position  and  master  2  sustains  the  full  load  of  the  beam. 


Figure  13.  Beam  angle  for  internet  test  (b=40).  Figure  14.  Vertical  force  command for  internet  test. 

To  demonstrate  the  feasibility  of  the  wave  variable  approach  for  longer  time  delays,  an  internet  delay 
simulator  was  used  to  generate  0.5  sec  and  1  sec  roundtrip  time  delays  as  shown  in  Figures  15  and  16, 
respectively.  The  decrease  in  system  stiffness  from  the  internet  test  is  evidenced  by  the  lower  frequency 
oscillations.  In  addition,  the  apparent  mass  at  the  handle  increases  from  approximately  0.5  kg  for  a  50  ms 
roundtrip  delay  to  10  kg  and  20  kg  for  roundtrip  delays  of  0.5  and  1  sec,  respectively.  The  heavier  feel  of  the 
handle  also  made  it  more  difficult  for  the  operator  to  control  contributing  further  to  the  degradation. 
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6.  CONCLUSIONS 

Internet  experiments  conducted  thus  far  indicate  the  feasibility  of  conducting  both  remote  assessment  and 
cooperative  rehabilitation  over  the  internet  using  robotic  devices.  During  a  cooperative  internet  task  between 
robots  500  miles  apart,  time-delays  of  up  to  110  ms  produced  borderline  instability  without  compensation. 
However,  under  wave  variable  control,  the  system  was  robust  to  time-delays,  and  there  was  an  almost 
imperceptible  increase  in  the  apparent  mass  of  the  handle.  Packet  loss  was  found  to  be  less  than  1%  at 
transfer  rates  of  100  Hz  when  using  UDP  transmission. 

We  are  currently  testing  even  larger  time-delays  and  examining  several  other  cooperative  tasks  such  as 
rowing  and  air  hockey.  We  are  also  engaged  in  a  cooperative  effort  with  the  National  Rehabilitation  Hospital 
in  Washington,  DC  to  test  stroke  patients  over  the  internet  and  apply  standard  metrics  for  assessment.  In 
addition,  a  head-mounted  display  and  tracker  are  being  integrated  into  the  system  to  allow  for  more  realistic 
simulations  using  3D  visualization.  Coordination  of  the  haptic  and  visual  feedback  in  the  simulator 
(stereopsis)  is  an  area  of  ongoing  research  as  are  strategies  for  dealing  with  packet  loss  during  less  reliable 
transmission. 
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Robotic  rehabilitation  over  the  internet 


Figure  1.  A  researcher  engages  in  a  game  of  air  hockey  on  an 
lnMotion2  robot. 


Robots  have  been  explored  as  possible 
rehabilitation  aids  in  laboratory  settings 
for  well  over  a  decade.  These  investi¬ 
gations  have  recently  expanded  into  the 
field  of  ‘teletherapy’  whereby  a  clini¬ 
cian  can  interact  with  patients  in  remote 
locations.  However,  time-delays  en¬ 
countered  in  force  feedback  between 
remotely-located  rehabilitation  devices 
can  cause  instability  in  the  system,  and 
have  so  far  prevented  widespread  ap¬ 
plication  over  the  internet. 

The  ISIS  Center  at  Georgetown  Uni¬ 
versity  Medical  Center  has  recently 
assembled  a  robot  rehabilitation 
testbed  consisting  of  a  pair  of 
InMotion2  robots'  connected  over  the 
internet  (see  Figure  1).  The  specific 
aims  of  our  research  are  twofold:  to 
enable  a  clinician  to  assess  the  physi¬ 
cal  condition  of  a  patient’s  arm  using  metrics 
such  as  strength,  dexterity,  range  of  motion,  and 
spasticity;  and  to  help  a  clinician  perform  co¬ 
operative  rehabilitative  tasks  with  a  patient  us¬ 
ing  a  virtual  environment  intended  to  simulate 
active  daily  living  (ADL)  tasks. 

An  example  of  a  cooperative  rehabilitation 
task  is  depicted  in  Figure  2.  The  patient  and 
therapist  ‘pick  up’  opposite  ends  of  a  virtual 
beam  by  grasping  the  handle  at  the  end  of  the 
robot  or  ‘master’  arm.2  The  mass  parameters 
of  the  object  such  as  mass,  length,  and  inertia 
can  be  adjusted  to  correspond  to  real-life  ob¬ 
jects  using  a  graphical  user  interface  (GUI)  on 
the  robot’s  computer.  As  the  object  is  being 
‘lifted’,  the  side  that  is  lower  will  begin  to  feel 
more  of  the  weight,  thus  stimulating  the  par¬ 
ticipants  to  maintain  the  beam  in  a  horizontal 
position.  Also,  if  one  side  tugs  on  the  object, 


the  other  side  feels,  it  thus  encouraging  a  co¬ 
operative  strategy  to  lift. 

Time-delay  compensation  is  key  to  avoid¬ 
ing  instabilities  in  force-feedback  interaction 
over  a  distance.  In  our  research,  the  wave- vari¬ 
able  method  emerged  as  the  most  natural  ap¬ 
proach  for  manipulating  virtual  objects  over  the 
internet.3  Instead  of  sending  force  commands 
directly,  an  impedance  ‘wave’  command  is  is¬ 
sued  by  the  master  and  then  subsequently  re¬ 
flected  by  the  virtual  object  or  ‘slave’.  How 
much  of  the  wave  is  reflected  depends  upon 
the  impedance  of  the  virtual  object:  a  compli¬ 
ant  object  will  not  reflect  the  incoming  wave 
as  greatly  as  a  rigid  wall. 

The  robots  are  controlled  by  computers  run¬ 
ning  RT-Linux  and  connected  to  the  internet 
as  illustrated  in  Figure  3.  When  controlling  a 
patient’s  robot  directly,  a  client  application  on 


that  patient’s  computer  connects  to  a 
server  on  their  therapist’s  computer. 
User  Datagram  Protocol  (UDP)  en¬ 
abled  communication  speeds  of  up  to 
1 00  1 6-byte  datasets  per  second.  A  vir¬ 
tual  object  generator  (V OG)  calculates 
the  motion  of  the  object  resulting  from 
the  force  inputs  from  the  patient  and 
the  therapist.4  An  artificial  time  delay 
(T1  in  Figure  3)  is  introduced  between 
the  VOG  process  and  the  local  control¬ 
ler  to  maintain  symmetry  in  communi¬ 
cation  between  the  VOG  and  the  two 
robots. 

To  date,  we  have  successfully  con¬ 
ducted  the  virtual  beam  task  between 
Georgetown  University  and  Interactive 
Motion  Technology  in  Cambridge, 
Massachusetts.  Internet  delays  of  up  to 
110ms  were  observed  which  caused 
borderline  stability  without  compensation.  The 
wave-variable  controller  was  found  to  be  ro¬ 
bust  to  variations  in  time  delay,  and  perfor¬ 
mance  degraded  only  gradually  with  increas¬ 
ing  delay.  The  degradation  manifests  itself  as 
a  higher  perceived  inertia  at  the  handle  which 
was  still  quite  manageable  even  with  artifi¬ 
cially-induced  delays  of  over  Is. 

We  are  currently  testing  even  larger  time 
delays  and  examining  the  incorporation  of 
sensed  force  into  the  wave-variable  methodol¬ 
ogy  to  produce  higher-bandwidth  feedback  to 
the  clinician.  In  addition,  a  head-mounted  dis¬ 
play  and  tracker  are  being  integrated  into  the 
system  to  allow  for  more  realistic  simulations 
using  three-dimensional  visualization.  Coordi¬ 
nation  of  the  haptic  and  visual  feedback  in  the 
simulator  is  an  area  of  ongoing  research. 

Continues  on  page  10. 
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Figure  2.  A  virtual  beam  task  realized  with  two  robot  handles  coincident 
with  the  ends  of  the  beam. 


Figure  3.  System  architecture  for  implementing  a  cooperative  internet 
task. 
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Enhanced  haptic  sensitivity  in  surgical  telemanipulation 

Continued  from  page  12. 


implementation  of  enhanced  sensitivity  in  the 
desired  stiffness  range. 

Figure  2  shows  some  experimental  results, 
where  an  elastic  rubber  band  was  stretched  us¬ 
ing  a  one-dimensional  teleoperation  system. 
The  position-force  curve  shows  a  changing 
slope,  corresponding  to  a  changing  local  stiff¬ 
ness.  Thereby,  two  major  areas  can  be  assigned. 
At  first,  when  the  band  is  only  slightly 
stretched,  a  stiffness  of  about  220N/m  is  de¬ 
tected.  Further  stretched,  the  stiffness  drops  to 
about  170N/m.  The  haptic  device  used  shows 
clear  Coulomb  friction,  which  adds  damping, 
but  does  not  alter  the  local  stiffness  character¬ 
istics.  The  fact  that  the  slopes  of  the  curves  in 
the  two  areas  differ  more  at  the  master  side  than 
at  the  environment,  shows  how  the  stiffness  dif¬ 
ference  between  the  two  areas  is  ‘blown  up’. 
In  fact,  the  difference  is  such  that  the  areas  eas¬ 
ily  can  be  distinguished.  This  enhancement  of 
discriminability  between  tissues  could  enable 
surgeons  to  more  securely  define  tumor  bound¬ 
aries,  and  have  a  higher  confidence  on  whether 
crucial  structures  in  the  neighborhood  are  clean 


or  in  an  infected  zone.  And,  as  surgeons  are 
able  to  feel  better,  safety  will  increase. 
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Figure  2.  Experimental  results  on  the  interaction  with  a  rubber  band:  the  environment  (grey),  and  the 
positions  and  forces  at  the  interface  to  the  human  operator  (black).  The  slope  of  the  position-force  curves 
denotes  the  local  stiffness. 
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Bone  Subtraction  CTA  Techniques  for  Skull  Base 
Vascular  Related  Pathology:  Initial  Clinical  Experience 
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Georgetown  University  Medical  Center 


Purpose 

Assessing  bone  subtraction  CTA  (bsCTA)  technique  in  determining  intracranial  vascular 
pathology  within  and  adjacent  to  skull  base  osseous  structures. 

Approach/Methods 

We  selected  patients  who  have  possible  vascular  pathology  at  the  skull  base  level  to 
perform  bsCTA.  A  non-contrast  and  contrast  enhanced  CT  scans  were  sequentially 
performed  with  identical  scanning  protocols  on  a  4-slice  Multidectector  CT.  A 
commercial  software  (Infinitt  Rapidia  ,  Seoul,  Korea),  was  used  to  register  and  subtract 
these  two  acquired  volumes  resulting  in  generation  of  a  third  subtracted  vessel-only 
volume.  This  subtracted  volume  was  then  imported  into  a  dedicated  3D  workstation 
(TeraRecon,  California,  U.S.A)  for  final  multiple  volumetric  rendering  viewings. 

The  bone  subtraction  CTA  volume  dataset  was  then  compared  to  traditional  digital 
subtraction  angiography  (DSA),  the  current  gold  standard,  and  3D  rotational  DSA 
(rDSA)  for  image  correlation. 

Findings 

The  appreciation  of  the  aneurysm  morphologies  such  overall  size,  neck  origin,  and  neck 
size  are  better  seen  when  bsCTA  technique  is  applied  to  intracranial  CTA.  These 
important  parameters  show  similar  image  correlation  on  bsCTA  as  they  are  on  the  3D 
rDSA  images. 

Conclusions 

Traditional  non-subtracted  CTA  is  limited  at  the  level  of  skull  base  where  osseous 
structures  are  the  predominant  factor.  The  bsCTA  technique  enhances  visualization  of 
vascular  pathology  within  this  level.  This  subtraction  technique  is  robust,  operator  non¬ 
dependent,  and  requires  little  additional  post-processing  effort.  Most  importantly,  it  adds 
additional  diagnostic  certainty  to  our  noninvasive  CTA  armamentarium. 
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Abstract 

Rationale  and  Objectives.  To  evaluate  the  feasibility  of  using  a  joystick-controlled 
robotic  needle  driver  under  CT  fluoroscopy  for  lung  biopsy  in  a  phantom  study. 

Materials  and  Methods.  A  respiratory  motion  phantom  incorporating  cranial-caudal 
motion  of  a  foam  liver  phantom  was  developed.  Synthetic  lesions  of  about  1 .5  mm  in 
diameter  were  created  using  an  agar  mix.  CT  fluoroscopy  was  used  to  visualize  the 
motion  of  a  lesion  in  and  out  of  the  scan  plane.  The  phantom  could  be  paused  on  demand 
for  up  to  30  seconds  to  simulate  a  breath  hold.  A  joystick-controlled  robotic  needle 
driver  was  used  by  a  radiologist  experienced  in  lung  biopsy  to  drive  an  1 8  gauge  needle 
on  the  scan  plane  toward  the  lesion. 

Results.  The  radiologist  was  able  to  hit  the  lesion  on  all  20  attempts.  The  average  time  to 
drive  the  needle  was  12.1  seconds  with  a  standard  deviation  of  3.1  seconds.  The  average 
tube  current-time  product  was  955  mAs  with  a  standard  deviation  of  1 80  mAs.  The 
average  dose-length  product  was  92.5  mGycm  with  a  standard  deviation  of  42  mGycm. 

Discussion.  A  joystick-controlled  robotic  needle  driver  can  be  used  to  accurately  target 
lesions  in  a  respiring  phantom  under  CT  fluoroscopy.  This  technique  may  allow  the 
radiologist  to  more  precisely  manipulate  the  needle  with  much  less  radiation  exposure. 
Additional  studies  are  planned  to  further  automate  this  process  and  to  investigate  the  role 
of  robotic  systems  for  lung  biopsy  in  the  clinical  area. 
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1  Introduction 

The  use  of  CT  for  lung  cancer  screening  is  rapidly  expanding.  This  screening  has 
revealed  a  large  number  of  small  pulmonary  nodules.  Most  of  these  nodules  are  benign 
and  less  than  1  cm  in  size.  They  are  often  dismissed  as  clinically  irrelevant,  sometimes 
inappropriately.  Some  recent  reports  suggest  that  a  higher-than-previously-suspected 
portion  of  these  small  pulmonary  nodules  are  malignant;  for  example,  lung  carcinoma 
rates  of  58%  were  reported  in  a  group  of  patients  that  had  thoracoscopic  biopsies  of 
nodules  less  than  1  cm  in  diameter[l].  It  is  therefore  becoming  increasingly  important  to 
obtain  a  tissue-specific  diagnosis  of  these  small  lesions. 

The  diagnostic  accuracy  of  transthoracic  needle  biopsy  (TNB)  is  not  ideal.  Yankelevitz 
et  al.  state  that  needle  misplacement  probably  represents  the  single  most  common  cause 
for  a  false-negative  biopsy[2].  A  needle  trajectory  misalignment  of  only  3  degrees  over  a 
distance  of  10  cm  will  result  in  a  deviation  substantial  enough  to  miss  a  1  cm  nodule[2]. 
In  addition,  the  needle  bevel  causes  the  needle  to  deflect  as  it  is  advanced,  and  thus 
requires  either  anticipation  of  the  degree  of  deviation,  or  repositioning  of  the  needle 
placement.  Although  most  patients  with  benign  disease  will  be  identified  correctly,  false 
negative  diagnoses  occur,  usually  due  to  sampling  error[3].  Missing  the  lesion  even  by  a 
fraction  of  a  millimeter  precludes  adequate  sampling[2].  Additionally,  the  diagnostic 
accuracy  of  TNB  drops  off  precipitously  for  small  nodules[4].  This  problem  reveals  the 
importance  of  path  planning  under  CT  guidance  as  it  is  presently  performed. 

White  et  al.  explain  that  a  standard  CT  has  several  limitations  that  are  a  consequence  of 
its  lack  of  real-time  visualization^].  It  may  take  multiple  attempts  to  localize  the  nodule 
and  achieve  proper  angulation  of  the  needle  toward  the  nodule.  Additionally,  respiratory 
motion  may  alter  the  anatomic  relationships  with  respect  to  the  nodule,  and  compensation 
can  be  difficult  in  the  absence  of  real-time  observation[5].  In  the  conventional  CT 
gantry,  the  actual  sampling  of  the  lesion  and  the  location  of  the  needle  tip  are  not 
observed  in  real-time[5],  a  factor  that  is  essential  for  a  successful  biopsy[6].  Therefore, 
CT  fluoroscopy,  a  relatively  new  modality,  is  gaining  acceptance  in  image  guidance  for 
TNB. 

The  advent  of  CT  fluoroscopy  combines  the  advantages  of  both  CT  and  fluoroscopy  in 
that  images  can  be  viewed  in  nearly  real-time[7].  Lesions  with  CT  fluoroscopy  can  be 
accessed  even  if  they  lie  in  less  favorable  locations  and  are  smaller  in  size.  The 
procedure  can  also  be  performed  more  quickly  and  in  less  cooperative  patients[8].  CT 
fluoroscopy’s  biggest  advantage  may  be  that  the  location  of  the  needle  tip  can  be 
precisely  ascertained  as  the  nodule  is  being  aspirated,  and  thus,  the  number  of  needle 
passes  can  be  reduced  [7]. 

CT  fluoroscopic  guidance  may  facilitate  placement  of  needles  in  the  edge  of  larger 
lesions  and  thus  avoid  an  equivocal  biopsy  of  the  necrotic  center.  Additionally,  if  real¬ 
time  imaging  in  CT  fluoroscopy  is  used,  it  is  possible  to  see  the  needle  push  away  the 
nodule  (balloting)  rather  than  pierce  it.  A  modification  in  technique  can  be  immediately 
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employed  using  a  more  rapid  puncture  to  pierce  the  specimen[5].  There  is  promising 
preliminary  experience  using  CT  fluoroscopy  for  biopsies  of  nodules  less  than  1 .5  cm  in 
diameter,  with  diagnostic  tissue  obtained  in  94%  of  the  cases[5]. 

However,  the  use  of  CT  fluoroscopy  for  lung  biopsy  and  other  procedures  has  not 
expanded  as  rapidly  as  one  might  expect.  An  important  safety  concern  raised  by  this 
technology  is  the  radiation  that  the  radiologists’  hands  are  exposed  to  during  their 
attempts  to  redirect  or  reposition  the  needle[9].  Therefore,  needle  holders  have  been 
developed  and  their  comparative  use  in  CT  fluoroscopy  has  been  evaluated[10,  11]. 
Alternatively,  another  method  of  avoiding  the  primary  beam  is  to  use  a  surgical  clamp  to 
grasp  and  guide  the  needle  in  real-time[5].  However,  none  of  these  techniques  would 
completely  remove  the  radiologist  from  the  ionizing  radiation;  moreover,  manipulation 
and  accurate  alignment  of  a  needle  with  an  off-centered  clamp  or  needle  holder  can  be 
difficult. 

To  minimize  the  radiation  exposure  to  the  radiologist,  and  to  provide  a  steady  platform 
for  needle  guidance  and  placement,  we  have  been  experimenting  with  a  joystick- 
controlled  needle  driver  robot  under  CT  fluoroscopy.  This  robot  was  originally  developed 
by  the  Urology  Robotics  (URobotics)  Laboratory  at  Johns  Hopkins  Medical  Institutions 
for  percutaneous  renal  access  under  fluoroscopy[12].  A  more  recent  model  of  the  robot 
was  employed  at  Georgetown  University  Hospital  for  a  clinical  trial  of  spinal  blocks[13]. 
A  photograph  of  the  robot  is  shown  in  Figure  1 .  The  robot  consists  of  a  touch  screen  and 
joystick  for  control,  a  mechanical  arm,  a  needle  driver,  and  a  mounting  base[14]. 

This  paper  describes  our  initial  investigations  with  this  device  under  CT  fluoroscopy  in  a 
phantom  study.  The  purpose  of  the  phantom  study  was  to  verify  the  ability  of  a 
radiologist  to  use  the  robotic  needle  driver  to  hit  lesions  in  a  synthetic  lung.  The  long 
term  goal  of  our  work  is  to  develop  an  integrated  system  for  robotic  lung  biopsy. 


2  Materials  and  Methods 

The  system  concept  is  shown  in  Figure  2.  The  real-time  CT  fluoroscopy  image  is 
captured  using  a  frame  grabber  card  (Accustream  170,  Foresight  Imaging,  Lowell, 
Massachusetts,  USA).  The  frame  grabber  is  a  commercial  product  that  occupies  a  spare 
slot  in  the  computer  backplane  of  the  robot  controller  and  allows  the  system  to  capture 
the  CT  images  in  real-time.  This  real-time  image  capture  is  essential  for  our  long  term 
goal  of  detecting  the  position  of  the  target  lesion  automatically  from  the  image.  If  the 
target  lesion  can  be  automatically  detected,  we  can  then  command  the  robot  to 
automatically  drive  the  needle  toward  the  lesion. 

A  picture  of  the  experimental  setup  is  shown  in  Figure  3.  The  study  was  done  in  the  CT 
room  at  Georgetown  University  Medical  Center  using  a  Siemens  Somatom  Volume 
Zoom  CT  scanner.  A  custom-designed  respiratory  motion  phantom  incorporating  a 
synthetic  lung  was  developed  for  this  study.  The  robotic  needle  driver  was  mounted  on 
the  CT  table  using  a  specially  constructed  mounting  bracket.  The  bracket  incorporated 
straps  that  ran  under  the  CT  table  to  hold  the  robot  tight. 
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The  respiratory  motion  phantom  is  shown  in  Figure  4.  The  phantom  consists  of  a  torso 
model,  a  rib  cage  taken  from  an  anatomical  bone  model,  a  rubber  like  skin  layer  (Limbs 
and  Things,  Bristol,  UK),  and  a  synthetic  lung.  The  synthetic  lung  was  molded  from  a 
two-part  flexible  foam  (FlexFoam  II,  Smooth-On,  Easten,  PA)  using  aplastic  lung  model. 
The  respiratory  motion  phantom  includes  a  one  degree  of  freedom  motion  platform  that 
simulates  cranial-caudal  motion  and  can  be  programmed  from  a  laptop  computer.  The 
synthetic  lung  was  attached  to  the  radiolucent  portion  of  the  motion  platform. 

Synthetic  lesions  were  created  by  mixing  agar  and  injecting  it  into  the  lung  using  a 
syringe.  This  technique  enabled  us  to  control  the  size  of  the  lesion  with  some  precision.  A 
lesion  of  about  1.5  cm  was  created  as  shown  in  Figure  5. 

The  robot  was  then  positioned  so  that  the  needle  was  aligned  with  the  CT  scan  plane 
(Figure  6).  The  respiratory  motion  phantom  was  positioned  so  that  the  lung  lesion  would 
move  in  and  out  of  the  scan  plane.  The  respiratory  motion  phantom  was  activated  with  a 
respiratory  rate  of  about  15  breaths  per  minute  and  an  excursion  distance  of  about  1.5  cm. 
The  interventional  radiologist  (CW)  activated  the  CT  fluoroscopy  imaging  mode  by 
stepping  on  the  foot  panel  and  watching  the  image  on  the  in-room  monitor  (Figure  7). 
When  the  lung  lesion  moved  into  the  scan  plane,  a  button  was  pressed  on  the  respiratory 
motion  controller  to  pause  the  respiratory  motion  for  up  to  30  seconds,  simulating  a 
patient  breath  hold.  The  interventional  radiologist  then  used  the  joystick  to  command  the 
robot  to  drive  the  needle  toward  the  lesion.  A  total  of  20  trials  were  done. 


3  Results 

The  results  of  the  study  are  given  in  Table  1 .  The  radiologist  was  able  to  hit  the  lesion  on 
all  20  trials.  The  average  time  to  drive  the  needle  was  12.1  seconds  with  a  standard 
deviation  of  3.1  seconds.  The  average  tube  current-time  product  was  955  mAs  with  a 
standard  deviation  of  180  mAs.  The  average  dose-length-product  (DLP)  as  a  measure  for 
the  applied  radiation  dose  was  92.5  mGycm  with  a  standard  deviation  of  42  mGycm.  A 
CT  fluoroscopy  image  showing  the  needle  in  the  lung  lesion  is  shown  in  Figure  8. 
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Table  1:  Study  Results 


Trial 

Needle 

Recorded 

Recorded 

number 

driving 

tube 

dose 

time 

current 

length 

product 

seconds 

mAs 

mGycm 

1 

22 

1232 

50 

2 

11.3 

1369 

150 

3 

14 

1256 

160 

4 

15 

991 

90 

5 

11 

1008 

200 

6 

13 

1079 

75 

7 

10.5 

867 

25 

8 

12 

867 

125 

9 

9.5 

1085 

75 

10 

14 

961 

100 

11 

8 

743 

50 

12 

12.5 

932 

70 

13 

12 

1003 

80 

14 

13 

914 

100 

15 

9 

719 

100 

16 

14 

920 

100 

17 

9 

843 

100 

18 

9 

749 

50 

19 

11 

767 

50 

20 

12 

796 

100 

Average 

12.1 

955 

92.5 

Standard 

deviation 

3.1 

180 

42 

4  Discussion 

Based  on  the  study  data  reported  here,  it  is  feasible  to  use  a  radiologist-controlled  robotic 
needle  driver  to  accurately  place  needles  during  breath  holds  of  a  lung  phantom  with 
simulated  respiratory  motion.  This  discovery  is  a  first  step  towards  developing  a  fully 
automated  robotic  system  for  CT  fluoroscopy  guided  lung  biopsy.  All  20  trials  were 
completed  without  difficulty  and  no  system  failures  were  observed  using  the  robot.  The 
robot  provides  a  steady  and  precise  holder  for  the  needle  and  is  capable  of  keeping  the 
needle  on  the  CT  scan  plane  so  that  the  procedure  can  be  readily  visualized  by  the 
radiologist  while  minimizing  his  or  her  exposure  to  radiation.  The  radiologist  can  view 
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the  location  and  trajectory  of  the  needle  as  it  is  directed  toward  the  lesion  in  real-time, 
since  the  radiologist’s  hand  is  not  near  the  path  of  the  x-ray  beam. 


Other  methods  to  aid  in  needle  guidance  have  been  proposed,  such  as  needle  holders  and 
a  CT-integrated  stereotactic  arm  (PinPoint,  Philips  Medical  Systems,  Cleveland,  Ohio). 
However,  these  methods  do  not  give  the  same  degree  of  precision  and  incremental 
motion  as  the  actively  driven  robot  tested  here. 

From  the  clinical  perspective,  the  current  approach  to  percutaneous  lung  biopsy 
procedures  is  largely  empiric.  The  radiologist  uses  images  from  CT  scanning  to  estimate 
the  best  approach  to  the  lesion  that  avoids  structures  such  as  ribs  and  allows  entrance  into 
the  lung  lesion.  If  this  estimate  is  incorrect  and  the  lesion  is  missed,  the  needle  tip  may 
need  to  be  readjusted,  increasing  procedure  time  and  the  risk  of  complications  such  as 
pneumothorax.  The  real-time  feedback  provided  by  CT-fluoroscopy  is  valuable  in 
permitting  quick  adjustment  of  needle  position  but  this  advantage  is  somewhat  mitigated 
by  the  need  to  avoid  exposure  of  the  operator’s  hands  during  the  biopsy.  Most 
radiologists  currently  use  an  “adjust  and  peek”  approach  to  needle  adjustment  rather  than 
directly  visualizing  needle  movement  with  CT  fluoroscopy.  The  joystick-controlled 
robotic  needle  driver  obviates  radiation  to  the  radiologist  and  may  facilitate  real-time  CT 
fluoroscopic  biopsies. 

The  next  step  in  our  research  program  is  to  further  automate  the  biopsy  process.  We  have 
been  developing  an  algorithm  to  automatically  track  a  lesion  under  CT  fluoroscopy  and 
then  command  the  robot  to  drive  the  needle  to  the  lesion  [15].  Phantom  studies  have  been 
done  using  this  algorithm  and  we  have  begun  swine  studies  under  an  approved  animal 
protocol. 
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Figure  Captions 

Figure  1 :  Robot  showing  touch  screen,  translational  mechanism,  and  needle  driver  end- 
effector 

Figure  2:  System  concept  for  robotic  lung  biopsy 

Figure  3:  Experimental  setup  showing  robot,  phantom,  and  needle  on  scan  plane 

Figure  4:  Respiratory  motion  phantom  with  motor  at  left  end  and  lung  at  right  end 

Figure  5:  Axial  CT  image  of  agar  injection  to  create  synthetic  lesion  in  lung 
(lesion  size  is  1.44  cm  as  measured  on  the  screen) 

Figure  6:  Close-up  view  showing  robot,  phantom,  and  needle  held  on  CT  scan  plane 
(note  red  laser  light  from  CT  scanner) 

Figure  7:  Radiologist  watching  monitor  and  manipulating  robot  with  joystick 
Figure  8:  Axial  CT  image  of  needle  in  lung  lesion 
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Figure  1:  Robot  showing  touch  screen,  translational  mechanism,  and  needle  driver 
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Figure  2:  System  concept  for  robotic  lung  biopsy 
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Figure  3:  Experimental  setup  showing  robot,  phantom,  and  needle  on  scan  plane 
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Figure  5:  Axial  CT  image  of  agar  injection  to  create  synthetic  lesion  in  lung 
(lesion  size  is  1.44  cm  as  measured  on  the  screen) 


Figure  6:  Close-up  view  showing  robot,  phantom,  and  needle  held  on  CT  scan  plane 

(note  red  laser  light  from  CT  scanner) 
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Figure  7:  Radiologist  watching  monitor  and  manipulating  robot  with  joystick 


Figure  8:  Axial  CT  image  of  needle  in  lung  lesion 
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ABSTRACT 

This  paper  describes  a  method  for  translational  and  orientational  alignment  of  a  robotic  needle  driver  based  on  image 
servoing  and  x-ray  fluoroscopy.  The  translational  process  works  by  segmenting  the  needle  in  a  frame-grabbed 
fluoroscopic  image  and  then  commanding  the  robot  to  automatically  move  the  needle  tip  to  the  skin  entry  point.  The 
orientational  alignment  is  then  completed  based  on  five  different  positions  of  the  needle  tip.  Previously  reported 
fluoroscopy  servoing  methods  use  complex  robot-image  registration  algorithms,  fiducial  markers,  and  two  or  more 
dissimilar  views  that  included  moving  the  fluoroscope.  Our  method  aligns  the  needle  using  one  setting  of  the 
fluoroscope  so  that  it  does  not  need  to  be  moved  during  the  alignment  process.  Sample  results  from  both  the 
translational  and  orientational  steps  are  included. 

Keywords:  fluoroscopy  servoing,  needle  driver  robot,  needle  segmentation,  medical  robotics 


I.  INTRODUCTION 

Minimally  invasive  procedures  are  becoming  more  widespread  because  they  reduce  patient  trauma  and  improve 
recovery  time.  To  successfully  perform  these  procedures,  physicians  need  as  much  visualization  information  as  possible. 
This  visualization  is  typically  achieved  through  different  imaging  techniques,  including  ultrasound,  fluoroscopy,  MRI, 
and  CT. 

We  have  recently  employed  a  robotic  arm  to  assist  physicians  in  needle  placement  for  minimally  invasive  spine 
procedures.1  The  robot  is  controlled  by  the  physician  using  a  joystick  and  thus  enables  the  physician  to  view  a 
fluoroscopy  image  in  real-time  while  manipulating  a  needle.  The  next  step  in  further  automating  this  procedure  is  to 
automatically  align  the  needle  with  respect  to  the  target  based  on  feedback  from  the  image. 

This  targeting  technique  is  called  “fluoroscopy  servoing”  and  has  been  explored  by  just  a  few  researchers.  Previous 
methods  have  used  complex  algorithms  for  robot-image  registration  and  two  or  more  dissimilar  views.  Our  approach  is 
“uncalibrated”  and  uses  a  two-phase  approach,  one  for  translation  and  one  for  orientation.  The  goal  of  the  translation 
stage  is  to  place  the  tip  of  the  needle  on  to  the  line  of  sight  that  passes  through  the  target.  After  this  step  is 
accomplished,  the  needle  is  aligned  with  the  line  of  sight  that  passes  through  the  target.  This  paper  presents  a  new 
technique  for  aligning  the  needle  with  the  target. 

Two  related  problems  in  visual  servoing  are  robustness  and  stability,  both  of  which  are  highly  desirable  in  medical 
procedures.  A  lack  of  robustness  results  primarily  from  inappropriate  detection  or  segmentation  of  the  target  or  the 
interventional  device,  such  as  the  surgical  needle. 

The  stability  of  visual  tracking  is  highly  dependent  on  the  ability  of  the  algorithm  to  process  the  visual  information 
quickly  enough  (in  real-time).  Often  a  dynamic  system  model  of  target  motion  is  used  to  estimate  the  target  location 
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within  the  image  and  a  region  of  interest  (ROI)  is  used  to  reduce  the  amount  of  image  data  processing.  This  technique 
has  proven  effective,  provided  the  ROI  is  sufficiently  large  to  detect  the  target  and  sufficiently  small  to  be  processed 
fast.  The  size  of  the  ROI  and  the  resolution  of  the  ROI  must  be  chosen  in  such  a  way  to  ensure  that  tracking  is  stable. 

The  main  advantages  of  our  new  alignment  method  are  speed  and  the  fact  that  the  C-Arm  does  not  have  to  move  during 
the  alignment  procedure.  The  speed  comes  from  the  fact  that  the  inputs  for  the  method  are  represented  by  five  different 
positions  of  the  head  of  the  needle  in  one  image  plane.  The  procedure  time  is  reduced  since  the  C-Arm  does  not  have  to 
move  during  the  alignment  process. 

The  first  attempts  to  perform  visual  servoing  using  X-ray  fluoroscopy  instead  of  CCD  (charge-coupled  device)  cameras 
were  done  by  Loser  and  Navab2  and  by  Patriciu  and  Stoianovici.3  All  previously  explored  methods  required  at  least  two 
dissimilar  views  of  the  target.  A  more  recent  approach  by  Mocanu4  required  just  one  view  for  the  alignment  procedure. 
Due  to  the  requirements  of  the  clinical  environment,  these  methods  have  to  provide  accuracy;  usually,  accurate  3D 
alignment  of  the  tool  with  respect  to  an  anatomic  target  is  achieved  and  an  estimate  of  the  required  insertion  depth  is 
offered.  Also,  a  few  earlier  studies  reported  computer-assisted  instrument  targeting  based  on  the  development  of 
specialized,  complex  robot-image  registration  algorithms.  These  methods  involved  fixed  target  locations  and  used  static 
images  of  fiducial  markers  to  estimate  the  robot-image  coordinate  mapping,  which  was  then  used  for  accurate  targeting. 
In  such  techniques,  planning  paths  provided  satisfactory  accuracy  since  there  was  no  significant  target  position 
displacement.  Such  methods  commonly  used  at  least  two  images  of  a  spatial  radio-opaque  marker  of  complex  geometry 
or  a  series  of  one-dimensional  marks  distributed  on  a  defined  pattern. 

All  image-guided  medical  procedures,  such  as  percutaneous  needle  access  or  radiofrequency  ablation,  require  targeting 
of  a  specific  instrument  at  an  exact  organ  location,  and  the  clinical  outcome  of  these  procedures  relies  on  targeting 
accuracy.  For  this  reason,  fluoroscopy  servoing  and  other  automated  techniques  should  continue  to  be  developed  for 
precision  targeting  applications.  As  the  benefits  of  medical  robotics  become  clearer,  they  may  be  more  intensively  used, 
in  particular  for  minimally  invasive  procedures. 


2.  METHODS  AND  MATERIALS 

The  system  described  in  this  paper  uses  a  needle  driver  robot  and  a  camera.  The  robotic  system  consists  of  a  radiolucent 
needle  driver  called  PAKY  (Percutaneous  Access  to  the  Kidney)  and  the  RCM  (Remote  Center  of  Motion)  robotic 
module.5  The  specialized  design  involved  the  addition  of  an  active  three  degrees  of  freedom  (3-DOF)  translational  stage, 
mounted  on  a  bridge-like  support,  linked  to  the  RCM  robot  by  a  7-DOF  passive  positioning  arm,  and  equipped  with  a 
central  braking  system,  which  can  be  easily  manipulated  at  desired  locations  and  firmly  locked  in  a  desired  position.6 

The  camera  simulates  the  C-Arm  imager  from  the  clinical  environment.  This  system  is  a  typical  example  of  an  image- 
based  servoing  system.  Images  from  the  camera  are  frame-grabbed  and  processed  in  real  time  on  the  robot  controller, 
where  all  of  the  image  processing  and  robot  commands  take  place. 

The  whole  process  consists  of  two  phases:  translation  and  alignment.  The  translation  process  will  place  the  tip  of  the 
needle  on  the  line  of  sight  that  passes  through  the  target.  The  alignment  stage  will  orient  the  needle  until  it  is  parallel  to 
the  line  of  sight  between  the  anatomical  target  and  the  projection  of  the  target  on  the  fluoroscopic  image. 


2.1.  Translation  stage 

At  the  beginning  of  the  translation  stage,  there  is  an  arbitrary  angle  between  the  needle  and  the  plane  of  the  fluoroscopic 
image.  The  anatomical  target  is  visible  in  the  image  field  of  view  and  the  needle  tip  is  located  at  an  arbitrary  distance 
from  the  anatomical  target.  The  distance  between  the  needle  tip  and  the  target  is  nevertheless  expected  to  be  small,  in 
the  range  of  a  couple  centimeters,  to  be  precise. 

The  initial  geometrical  configuration  of  the  system  is  illustrated  in  Figure  1.  The  coordinates  (x,y)  define  the  plane  in 
which  the  robot  arm  can  be  translated.  The  coordinates  (x’,y’)  define  the  reference  system  of  the  fluoroscopic  image 
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after  it  is  acquired  by  the  image  grabber.  The  translation  stage  is  used  to  bring  the  needle  tip  in  contact  with  the  line  of 
sight  between  the  anatomical  target  and  its  projection  in  the  fluoroscopic  image  plane,  as  illustrated  in  Figure  2. 


Figure  1.  Initial  geometrical  configuration 


Figure  2.  Configuration  at  the  end  of  alignment 


From  the  fluoroscopic  image  point  of  view,  the  needle  tip  will  be  in  contact  with  the  line  of  sight  to  the  target  when  the 
tip  is  seen  as  overlapping  the  target  on  the  image  plane  (Figure  2).  On  the  other  hand,  when  the  needle  tip  is  not  touching 
the  line  of  sight,  this  position  will  be  seen  in  the  fluoroscopic  image  as  the  tip  being  at  a  certain  distance  from  the 
anatomical  target  (Figure  1),  and  therefore,  there  will  be  a  vector  v’  relating  the  positions  of  the  needle  tip  and  the  target 
in  the  (x’,y’)  space. 

In  order  to  bring  the  needle  into  contact  with  the  line  of  sight,  the  robot  arm  must  be  displaced  in  the  plane  (x,y).  We 
must  therefore  convert  the  vector  v’  from  the  image  space  (x’,y’)  into  a  vector  v  in  the  robot  space  (x,y).  We  must  then 
find  a  transformation  suitable  for  mapping  (x’,y’)  coordinates  into  (x,y)  coordinates.  In  this  particular  geometrical 
configuration,  that  transformation  can  be  approximated  by  a  2D  affine  transform  if  we  ignore  the  perspective  effect  on 
the  formation  of  the  X-Ray  image.  The  error  introduced  by  this  approximation  is  lower  when  the  distance  from  the 
needle  tip  to  the  line  of  sight  is  small,  and  when  the  needle  tip  is  far  from  the  X-Ray  source.  A  two-dimensional  affine 
transform  is  fully  specified  by  6  coefficients.  Therefore,  if  we  know  the  (x,y)  and  (x’,y’)  coordinates  of  the  needle  tip  in 
at  least  6  different  positions,  we  are  able  to  estimate  the  parameters  of  the  2D  affine  transform.  Once  we  have  the  affine 
transform,  we  can  use  it  in  order  to  map  the  vector  v’  from  the  image  plane  into  a  displacement  v  in  the  robot  plane 
(x,y),  and  to  move  the  robot  arm  by  the  amount  required  to  put  the  needle  tip  in  contact  with  the  line  of  sight. 

Figure  3  illustrates  the  properties  of  the  robot  coordinate  system  (x,y)  when  it  is  mapped  into  the  fluoroscopic  image 
coordinate  system  (x’,y’).  An  affine  transform  maps  straight  lines  from  (x,y)  into  straight  lines  in  (x’,y’).  Angles  from 
the  (x,y)  space  are  not  conserved  when  mapped  into  the  (x’,y’)  space  because  the  image  plane  and  the  robot  plane  are 
not  parallel.  Figure  4  illustrates  the  selection  of  6  positions  used  to  estimate  the  coefficients  of  the  2D  affine  transform. 
These  positions  are  obtained  by  moving  the  robot  arm  in  the  (x,y)  space  by  known  amounts.  Translations  of  the  robot 
used  here  can  be  controlled  with  at  least  0.1  mm  precision.  The  locations  of  the  6  points  are  all  obtained  relative  to  an 
arbitrary  origin  by  applying  robot  translations  of  12  millimeters  each  time.  Thus,  we  acquire  7  positions  and  use  one  of 
them  as  the  coordinate  system  origin. 
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Figure  3.  Robot  coordinate  system  (x,y)  projected 
onto  fluoroscopic  image  space  (x',y') 


Figure  4.  Positions  used  for  affine 
transform  estimation 


Figure  7.  Maximum  Eigen  value 


Figure  8.  Projection  of  gradient 
onto  the  maximum  Eigen  vector 


Figure  9.  Population  of  ridge 
points  detected 


Figure  10.  Parametric  space  for 
detecting  ridge  points. 
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The  positions  of  the  needle  tip  in  the  image  plane  are  found  by  segmenting  the  needle  from  the  fluoroscopic  images.  A 
typical  fluoroscopic  image  is  presented  in  Figure  5,  where  the  needle  overlaps  a  section  of  the  patient’s  spine. 

The  process  used  for  segmenting  the  needle  is  illustrated  in 
Figure  11.  The  essential  aspect  of  this  segmentation  process  is 
the  detection  of  intensity  ridge  points  on  the  fluoroscopic 
image.  This  method  was  implemented  following  the  approach 
described  in  Eberly  20007  and  Ay  1  ward  20028.  The  image  is 
first  blurred  using  a  Gaussian  filter  that  produces  the  result 
shown  in  Figure  6.  From  this  blurred  image,  first  and  second 
derivatives  are  computed.  The  matrix  containing  all  the  second 
derivatives  of  intensities  at  one  point  corresponds  to  the 
Hessian  matrix  of  the  intensity  at  that  particular  location.  An 
Eigen  analysis  of  the  Hessian  matrix  leads  to  the  detection  of 
the  direction  along  which  the  second  derivative  has  a 
maximum  value,  as  well  as  the  value  of  such  second 
derivative.  This  value  appears  as  the  maximum  Eigen  value  of 
the  Hessian  matrix  as  illustrated  in  Figure  7.  The  scalar  product 
of  the  image  gradient  with  the  principal  eigenvector  provides  a 
measure  of  how  far  every  pixel  is  from  the  media!  line  of  the 
ridge,  as  illustrated  in  Figure  8. 


A  statistical  pattern  recognition  approach  is  implemented  by 
first  building  a  3D  parametric  space  where  a  point  is  placed  for 
every  pixel  in  the  image.  The  coordinates  of  the  point  in  the 
parametric  space  are  respectively  the  values  of  the  pixel  in  the 
blurred  image,  the  maximum  Eigen  value  image,  and  the 
gradient  projection  image.  The  population  of  points  in  this 
parametric  space  is  illustrated  in  Figure  10.  Points  belonging  to 
a  ridge  tend  to  cluster  together  in  the  region  represented  with  a 
green  pyramid.  When  the  image  pixels  associated  with  points 
in  that  region  are  highlighted  in  the  original  image,  we  obtain  a  set  of  ridge  points,  as  illustrated  in  Figure  9.  A  connected 
components  analysis  is  performed  for  this  set  of  points  with  the  aim  of  eliminating  small  sections  of  ridges  that  are 
commonly  found  in  anatomical  structures  such  as  vertebrae.  With  the  points  belonging  to  the  largest  connected 
component,  a  line  Hough  transform  is  computed  with  the  aim  of  finding  the  parameters  of  a  straight  line  parallel  to  the 
needle  in  the  image.  This  parametric  line  equation  allows  further  elimination  of  any  outliers.  The  remaining  points  are 
used  to  compute  the  position  of  the  needle  tip. 

The  segmentation  process  described  above  is  applied  7  times  in  order  to  find  each  one  of  the  needle  positions  illustrated 
in  Figure  4.  From  those  seven  segmentations,  six  relative  positions  are  computed  and  used  for  estimating  the  2D  affine 
transform  capable  of  mapping  the  robot  space  (x,y)  into  the  image  space  (x’,y’)  and  vice  versa.  The  affine  transform  is 
used  for  converting  the  vector  v’  from  the  image  into  the  vector  v  that  must  be  applied  to  the  robot  arm  in  order  to  place 
the  needle  tip  in  contact  with  the  line  of  sight  as  illustrated  in  Figure  2.  After  applying  such  translation  in  the  robot  arm, 
the  tip  of  the  needle  coincides  with  the  target  as  it  is  seen  in  the  fluoroscopic  image. 


Figure  1 1 .  Flow  diagram  of  the  segmentation 
method  and  needle  tip  position  estimation. 


2.2.  Alignment  stage 

In  this  stage,  the  physician  positions  the  C-Arm  to  obtain  a  good  view  of  the  target.  In  this  image,  the  tip  of  the  needle 
is  on  the  line  of  sight  that  passes  through  the  target  (as  described  in  Section  2.1).  From  this  point,  only  the  orientation 
stage  of  the  robot  is  used,  to  provide  a  remote  center  of  motion  capability.  This  ensures  that  the  tip  of  the  needle  is  fixed 
in  3D  and  the  needle  can  be  moved  in  two  planes  defined  by  the  kinematics  of  the  orientation  stage.  The  method  of 
aligning  the  needle  relies  on  a  few  basic  concepts. 
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Let  us  consider  the  needle  rotation  in  a  plane  around  its  tip,  as  illustrated  in  Figure  12.  If  the  image  plane  is  parallel  to 
the  rotation  plane,  then  the  trajectory  of  the  needle  head  is  a  circle  in  the  image  plane  (plane  A  in  Figure  12).  If  the 
planes  are  not  parallel,  then  the  needle  head  creates  an  ellipse  in  the  image  (plane  B  in  Figure  12). 


During  the  alignment  procedure,  we  have  access  only  to  the  data  that  the  image  plane  provides.  Consequently, 
estimation  of  this  ellipse  is  a  critical  to  the  needle  alignment  procedure  presented  here.  In  order  to  obtain  the  ellipse’s 
points,  the  coefficients  A,  B,  C,  D,  E,  and  F  from  the  following  general  second  degree  quadratic  equation  should  be 
estimated. 


A*x2  +  B*x*y  +  C*y2  +  D*x  +  E*y  +  F  =  0 


To  estimate  these  coefficients,  five  points  on  the  ellipse  are  needed.  These  points  are  obtained  with  some  human 
assistance,  meaning  the  physician  has  to  click  on  the  needle  head  in  the  image  five  times.  The  orientation  stage  of  the 
robot  is  used  to  move  the  needle  head  to  five  different  positions  in  a  plane.  After  each  move,  the  physician  must  identify 
the  needle  head  by  selecting  it  in  the  image  with  the  computer  mouse.  In  the  future,  an  automatic  procedure  for  tracking 
the  needle  head  would  be  desirable.  The  success  of  the  parameter’s  estimation  is  strongly  based  on  the  distribution  of 
the  five  points  that  are  used  for  fitting  the  ellipse.  These  points  should  be  evenly  distributed.  If  they  are  too  close,  the 
fitting  algorithm  may  not  estimate  the  parameters  precisely  enough. 

This  procedure  produces  the  coefficients  of  the  second  degree  quadratic  equation.  With  these  coefficients,  all  the  other 
points  of  the  ellipse  can  be  easily  computed.  Each  estimated  point  serves  as  an  input  for  the  next  phase  of  the  procedure. 
For  each  of  the  estimated  points,  the  following  steps  are  taken: 

1)  Having  the  point  on  the  ellipse  in  the  image  plane,  the  position  of  the  needle’s  head  in  space  may  be 
estimated; 

2)  Having  the  position  of  the  needle’s  head  in  space,  the  circle  described  by  the  head  when  rotating  in  the 
second  plane  of  the  RCM  may  be  estimated. 


presented  at  the  SPIE  Medical  Imaging  Symposium  2005 


3)  The  projection  of  the  circle  obtained  in  Step  2  is  also  an  ellipse  that  may  be  estimated.  If  this  ellipse 
represents  a  particular  case  representing  a  line,  it  means  that  the  ellipse’s  point  from  the  Step  1  is  the  one 
we  are  looking  for. 

The  alignment  method  described  above  also  takes  into  account  the  fact  that  the  orientation  stage  of  the  robot  can  rotate 
the  needle  in  two  planes  that  have  between  them  99.3  degrees  (fixed  by  design  of  the  orientation  stage).  The  alignment 
method  consists  of  two  steps  as  illustrated  in  Figure  13. 

1)  Rotate  the  needle  in  one  plane  (R0  plane)  into  a  position  that  makes  the  second  plane  (R1  plane) 
perpendicular  to  the  image  plane.  If  the  needle  is  moved  in  the  second  plane  after  this  step,  the 
needle  head’s  trajectory  is  a  line  that  represents  the  projection  of  the  needle  on  the  image  plane. 

2)  Move  the  needle  in  the  second  plane  so  that  it  becomes  a  point  in  the  image  plane.  After  this  step, 
the  needle  is  aligned  along  the  line  of  sight  that  passes  through  the  target. 


In  Figure  13,  three  positions  of  the  needle  are  shown.  The  needle  head  is  first  rotated  in  the  R0  plane  from  position  1  to 
position  2.  In  the  image  plane,  the  projection  P0  of  the  needle  head  that  corresponds  to  a  rotation  in  R0  plane  is  shown. 
If  the  plane  R0  is  parallel  to  the  image  plane,  then  the  projection  P0  would  be  a  circle.  Generally  speaking,  these  planes 
are  not  parallel,  and  consequently  the  projection  P0  is  an  ellipse.  The  best  initial  robot  configuration  makes  the  R0  plane 
as  parallel  as  possible  to  the  image  plane  so  that  the  ellipse  fitting  is  more  numerically  stable.  If  the  R0  plane  is  nearly 
perpendicular  to  the  image  plane,  then  the  projection  P0  will  become  a  degenerated  ellipse  and  close  to  a  line  segment. 
This  configuration  would  only  work  if  the  R0  plane  is  almost  perfectly  perpendicular  to  the  image  plane,  so  that  needle 
alignment  could  be  accomplished  by  rotation  in  the  R0  plane  only.  However,  we  prefer  to  avoid  this  situation  because  it 
may  lead  to  inconsistent  results  in  the  alignment  process. 
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The  purpose  of  rotating  the  needle  in  the  R0  plane  from  position  1  to  position  2  is  to  make  the  R1  plane  perpendicular  to 
the  image  plane.  If  this  positioning  is  accomplished,  then  the  projection  PI  of  the  needle  head  that  corresponds  to  a 
rotation  in  the  R1  plane  will  be  a  line  segment.  A  rotation  in  plane  R1  from  position  2  to  position  3  will  then  make  the 
needle  perpendicular  to  the  image  plane  so  that  it  becomes  a  point  in  the  image.  The  needle  will  then  be  aligned  with  the 
target  and  the  procedure  is  complete. 

The  algorithm  has  been  implemented  in  a  C++  class  called  EllipseEstimator.  This  work  was  initially  done  as  part  of  a 
project  to  develop  an  image-guided  surgery  software  toolkit.  The  code  was  archived  in  a  CVS  repository  and 
automatically  built  nightly  on  multiple  platforms.  This  environment  ensures  that  the  code  can  be  checked  out,  compiled, 
and  built  at  any  time.  A  testing  infrastructure  was  also  created  for  this  class.  Almost  each  public  method  has  a  testing 
procedure  that  makes  sure  the  code  performs  as  intended.  Tests  also  run  every  night  to  help  ensure  the  robustness  and 
quality  of  the  code. 

The  whole  procedure  may  be  described  by  the  following  sequence: 

1)  The  patient  is  positioned  on  the  procedure  table. 

2)  The  robot  is  positioned  on  the  table  over  the  patient  so  that  the  tip  of  the  needle  is  in  the  vicinity  of  the  skin  entry 
point. 

3)  An  anterior/posterior  fluoroscopic  image  is  obtained  such  that  the  target  can  be  clearly  seen  and  the  whole  needle 
is  in  the  image. 

4)  The  translational  alignment  procedure  described  in  Section  2.1  is  performed.  The  procedure  segments  and  tracks 
the  needle  and  moves  the  needle  to  the  precise  skin  entry  point. 

5)  The  orientational  alignment  procedure  described  in  Section  2.2  is  performed. 

6)  Once  the  needle  is  aligned,  a  lateral  fluoroscopic  image  is  used  to  control  the  depth  of  insertion. 


3.  RESULTS 

The  translation  procedure  was  run  in  the  lab  and  in  the  clinical  environment  with  good  results.9  The  alignment 
procedure  was  tested  in  the  laboratory  environment  using  a  standard  video  camera  and  a  frame  grabber.  The  orientation 
stage  was  moved  to  five  different  positions,  and  after  each  move  the  position  of  the  needle  tip  in  the  image  was  obtained 
by  selecting  it  with  the  mouse.  These  five  points  represent  the  input  data  for  the  EllipseEstimator  class.  As  output,  the 
class  produces  the  coefficients  A,  B,  C,  D,  E,  F  from  the  general  2nd  degree  equation.  With  these  parameters,  we 
computed  the  coordinates  of  points  that  belong  to  the  ellipse  at  the  pixel  level  and  plotted  all  these  points  on  the  screen. 

A  sample  result  is  shown  in  Figure  14.  In  this  figure,  the  R0  plane  of  the  RCM  is  almost  parallel  to  the  image  plane,  so 
the  projection  on  the  image  plane  is  almost  a  circle. 

A  second  example  is  shown  in  Figure  1 5.  In  this  figure,  more  of  an  ellipse  can  be  seen,  as  the  R0  plane  of  the  RCM  is 
not  as  parallel  to  the  image  plane. 

Both  tests  followed  the  standard  protocol.  The  alignment  process  started  with  the  tip  of  the  needle  coinciding  with  the 
target  marked  by  physician.  The  needle  was  automatically  positioned  in  five  different  spots  using  the  R0  stage  of  the 
RCM,  and  needle  head  positions  were  taken.  The  ellipse  parameters  and  the  points  that  belong  to  the  ellipse  were 
estimated  and  plotted  on  the  screen  with  red  dots,  as  can  be  seen  in  Figures  14  and  15. 
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Figure  14.  Sample  Result  #1 
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Figure  15.  Sample  Result  #2 
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A  human  observer  evaluated  the  results  and  determined  that  the  needle  head  accurately  followed  the  plotted  points  when 
rotated  using  R0  stage  of  RCM.  We  also  noted  that  the  needle  tip  remained  stationary  at  the  center  of  the  estimated 
ellipse.  Therefore,  we  believe  this  method  can  accurately  position  the  needle. 


4.  CONCLUSIONS 

A  targeting  method  for  fluoroscopy  servoing  based  on  a  new  alignment  procedure  has  been  presented.  The  alignment 
method  requires  only  a  single  C-arm  view  and  only  five  input  points.  Initial  laboratory  tests  showed  good  qualitative 
results.  The  next  step  would  be  to  do  phantom  testing  in  the  interventional  suite  and  verify  that  the  method  could  be 
used  for  accurate  needle  placement  under  fluoroscopy. 

The  new  method  is  fast  and  has  several  other  advantages.  There  is  no  need  for  pre-operative  registration,  bi-planar  X-ray 
images,  or  calibration  of  the  system  prior  to  use.  Currently,  the  method  is  semi-automatic,  since  the  physician  selects  the 
needle  head  in  the  images.  This  problem  can  be  solved  by  placing  a  marker  that  will  be  segmented  to  locate  the  needle 
tip.  The  fact  that  the  method  uses  the  principles  of  projective  geometry  makes  it  simple  and  accurate,  requiring  little 
human  interaction. 
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ABSTRACT 

An  effective  treatment  method  for  organs  that  move  with  respiration  (such  as  the  lungs,  pancreas,  and  liver)  is  a  major 
goal  of  radiation  medicine.  In  order  to  treat  such  tumors,  we  need  (1)  real-time  knowledge  of  the  location  of  the  tumor, 
and  (2)  the  ability  to  adapt  the  radiation  delivery  system  to  follow  this  constantly  changing  location.  In  this  study,  we 
used  electromagnetic  tracking  in  a  swine  model  to  address  the  first  challenge,  and  to  determine  if  movement  of  a  marker 
attached  to  the  skin  could  accurately  predict  movement  of  an  internal  marker  embedded  in  an  organ.  Under  an 
approved  animal  research  protocol,  an  electromagnetically  tracked  needle  was  inserted  into  a  swine  liver  and  an 
electromagnetically  tracked  guidewire  was  taped  to  the  abdominal  skin  of  the  animal.  The  Aurora  (Northern  Digital 
Inc.,  Waterloo,  Canada)  electromagnetic  tracking  system  was  then  used  to  monitor  the  position  of  both  of  these  sensors 
every  40  msec.  Position  data  from  the  sensors  were  then  analyzed  to  see  if  any  of  the  movements  showed  correlation. 
The  strongest  correlations  were  observed  between  external  anterior-posterior  motion  and  internal  inferior-superior 
motion,  with  many  other  axes  exhibiting  only  weak  correlation.  We  also  used  these  data  to  build  a  predictive  model  of 
internal  motion  by  taking  segments  from  the  data  and  using  them  to  derive  a  general  functional  relationship  between  the 
internal  needle  and  the  external  guidewire.  For  the  axis  with  the  strongest  correlation,  this  model  enabled  us  to  predict 
internal  organ  motion  to  within  1  mm. 

Keywords:  respiratory  motion,  radiotherapy,  radiosurgery,  image  guidance,  electromagnetic  tracking 


1.  INTRODUCTION 

An  effective  treatment  method  for  organs  that  move  with  respiration  (such  as  the  lung,  pancreas,  and  liver)  is  a  major 
goal  of  radiation  medicine.  Tumors  in  these  organs  have  typically  been  treated  using  a  variety  of  methods  to  account 
for  respiratory  motion.  The  first  and  simplest  of  these  methods  is  to  irradiate  the  tumor  plus  a  large  surrounding  area 
(margin)  in  order  to  ensure  that  the  moving  tumor  remains  within  the  region  of  prescribed  dose  at  all  times1.  Although 
this  method  ensures  sufficient  dose  to  the  tumor2,  it  also  tends  to  irradiate  a  large  amount  of  healthy  normal  tissue, 
increasing  the  likelihood  of  complications  or  the  creation  of  secondary  cancers  from  radiation  damage.  Another 
possible  approach  is  to  control  patient  respiration,  either  with  voluntary  breath-holding3,4  or  through  a  mechanical 
device  that  regulates  airflow  into  and  out  of  the  lungs5.  In  this  case,  the  treatment  beam  is  switched  on  once  breathing 
has  stopped  and  the  tumor  is  stationary.  This  method  can  significantly  reduce  the  margins  around  the  tumor,  but 
requires  a  high  level  of  patient  compliance  and  may  not  be  practical  for  patients  with  compromised  lung  capacity. 
Furthermore,  breath-holding  tends  to  significantly  extend  the  time  required  for  patient  treatment.  A  third  approach  is  to 
actively  monitor  the  respiration  and  turn  the  treatment  beam  on  during  a  particular  portion  of  the  respiratory  cycle6,7,8. 
This  method  requires  less  patient  compliance  than  breath-holding  and  has  similar  advantages  in  terms  of  the  ability  to 
reduce  tumor  margins;  however,  it  also  requires  a  much  longer  treatment  time  since  the  beam  is  only  on  for  a  short 
fraction  of  the  respiratory  cycle.  The  fourth  approach  is  to  monitor  the  respiration  and  actively  steer  the  radiation  beam 
to  follow  the  tumor  as  it  moves9,10.  Although  this  method  is  by  far  the  most  technically  demanding,  active  steering  can 
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potentially  achieve  the  tightly  conformal  margins  of  gating  or  breath-holding  without  significantly  extending  the 
treatment  time,  thus  creating  an  optimal  solution  for  respiratory  motion  compensation. 

The  active  steering  approach  to  respiratory  motion  compensation  requires  that  we  have  (1)  real-time  knowledge  of  the 
location  of  the  tumor,  and  (2)  the  ability  to  adapt  the  radiation  delivery  system  to  follow  this  constantly  changing 
location.  Although  a  variety  of  imaging  modalities  could  potentially  be  used  to  follow  the  tumor  location,  most  of  them 
are  not  practical  in  the  radiotherapy/radiosurgery  environment.  Ultrasound  cannot  effectively  penetrate  airspaces  within 
the  body  (such  as  the  lungs)  and  also  requires  a  skilled  operator,  who  obviously  cannot  be  standing  next  to  the  patient 
during  treatment.  Magnetic  resonance  imaging  (MRI)  systems  are  generally  too  confined  to  allow  much  access  to  the 
patient,  and  the  strong  magnetic  fields  would  likely  disrupt  the  operation  of  the  treatment  machine.  Computed 
tomography  (CT)  has  similar  geometric  limitations  to  MRI,  although  they  are  not  quite  as  prohibitive  since  the  CT 
scanner  tends  to  be  slightly  smaller.  Onishi  et  al.  reported  on  the  use  of  CT  in  a  radiation  therapy  environment", 
although  this  implementation  was  used  to  improve  patient  breath-holding  as  opposed  to  active  tumor  tracking. 
Fluoroscopy  has  a  fairly  small  footprint  and  can  generate  images  at  video  rates  (roughly  30  Hz),  and  has  been 
successfully  used  for  respiratory-gated  radiotherapy8,12.  However,  CT  and  fluoroscopy  both  place  an  additional 
radiation  dose  burden  on  the  patient. 

Since  none  of  these  imaging  techniques  are  truly  optimal,  there  has  been  much  interest  in  defining  some  other  means  to 
track  the  tumor  position  in  real-time.  Much  of  the  work  in  this  area  has  focused  on  the  use  of  optical  tracking  to 
measure  the  motion  of  skin  on  the  abdomen  or  thorax,  which  can  then  potentially  be  used  as  a  surrogate  signal  for 
internal  tumor  motion.  Optical  tracking  is  an  excellent  means  of  following  respiration  because  it  is  non-invasive,  has 
very  high  accuracy/precision,  is  relatively  inexpensive,  and  can  easily  provide  real-time  results.  However,  since  optical 
tracking  methods  depend  on  having  a  line-of-sight  to  the  target,  the  true  motion  of  the  internal  target  must  be  obtained 
through  some  other  means  and  then  correlated  to  the  skin  motion.  Some  investigators  have  used  fluoroscopy  with 
fiducial  markers  implanted  near  the  tumor  and  on  the  skin  surface,  so  that  a  single  fluoroscopy  image  measures  both 
internal  and  external  locations91314,  whereas  others  have  obtained  similar  data  using  MRI15.  These  studies  showed  that 
for  some  cases,  a  strong  correlation  existed  between  internal  tumor  motion  and  external  skin  motion.  However,  for 
other  cases,  the  correlation  was  rather  weak  or  was  negatively  affected  by  phase  differences  between  the  two  motions. 
Furthermore,  both  fluoroscopy  and  MRI  have  important  limitations.  Fluoroscopy  is  inherently  limited  to  two- 
dimensional  projections  and  thus  cannot  measure  three-dimensional  motion  (although  three-dimensional  motion  can  be 
obtained  from  multiple  simultaneous  projections).  Although  MRI  is  volumetric,  the  volume  is  typically  acquired  slice- 
by-slice,  so  there  is  always  the  potential  for  a  shift  in  the  anatomy  between  slices.  Even  with  a  respiratory-gated  pulse 
sequence,  one  cannot  be  certain  that  the  anatomy  always  returns  to  the  same  position  during  the  next  acquisition. 

In  this  paper,  we  report  on  the  use  of  electromagnetic  tracking  in  an  animal  model  to  obtain  data  on  the  correlation 
between  external  motion  and  internal  motion.  Although  electromagnetic  tracking  is  not  an  imaging  modality  and  is 
limited  to  following  discrete  points  in  a  three-dimensional  space,  it  has  distinct  advantages  over  other  methods.  First,  it 
does  not  use  ionizing  radiation,  and  thus  can  be  active  for  long  periods  of  time  without  any  adverse  effects  on  the 
patient.  Second,  it  tracks  continuously  and  in  three  dimensions,  so  we  are  able  to  capture  the  complete  temporal  and 
spatial  characteristics  of  the  motion.  Third,  the  system  itself  is  physically  compact,  and  thus  is  easily  able  to  be 
integrated  into  a  radiotherapy/radiosurgery  environment.  Finally,  unlike  optical  tracking  systems,  it  is  not  limited  to 
line-of-sight  operation.  Because  of  these  last  two  properties,  electromagnetic  tracking  could  conceivably  be  used  to 
directly  monitor  internal  tumor  motion;  we  have  an  active  research  effort  in  this  area. 

2.  METHODOLOGY 


2.1.  Electromagnetic  tracking  system 

For  this  study,  we  used  the  Aurora  (Northern  Digital  Inc.,  Waterloo,  Ontario,  Canada)  electromagnetic  tracking  system. 
The  system  consists  of  three  parts,  as  illustrated  in  Figure  1: 

(1)  a  tetrahedon-shaped  generator  that  creates  a  rapidly  switching  magnetic  field, 

(2)  specially  constructed  needles,  guidewires,  catheters,  or  other  tools  containing  tiny  wire  coils, 

(3)  a  control  unit  and  sensor  interface  units  that  measure  currents  induced  in  the  coils  by  the  changing  field. 
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Figure  1 :  Left  half  of  figure  shows  the  Aurora  System  from  Northern  Digital  Inc.  (field  generator  on  left,  control  unit  in  top  middle, 
sensor  interface  unit  on  right  side,  and  electromagnetically  tracked  probe  in  bottom  middle).  Right  half  of  the  figure  shows  sensor 
coils  and  tools  developed  and  manufactured  by  Traxtal.  Figures  courtesy  of  Northern  Digital  and  Traxtal. 

The  needles  and  guidewires  for  this  study  were  designed  and  manufactured  by  Traxtal  (Bellaire,  TX,  USA).  Needles 
were  18  gauge  (0.838  mm  nominal  outer  diameter)  and  the  guidewires  were  0.89  mm  diameter  (approximately  3 
French).  Once  the  system  is  activated,  it  continuously  reports  both  the  position  and  orientation  of  the  tools/probes  that 
are  plugged  into  the  interface  units.  In-house  software  is  used  to  poll  the  Aurora  and  read  out  position/orientation  data. 
The  overall  readout  frequency  is  between  10  and  25  Hz  depending  on  the  number  of  tools  that  are  plugged  into  the 
control  box. 

2.2.  Experimental  procedures 

All  research  in  this  study  was  carried  out  under  an  institutionally-approved  animal  protocol.  Small  (between  50  and  70 
pounds)  swine  were  anesthetized  and  positioned  supine  (ventral  side  up)  on  an  interventional  radiology  table.  The  table 
is  made  from  carbon  fiber  and  thus  is  radiolucent  and  nonmagnetic  while  still  providing  excellent  structural  strength. 

The  swine  were  positioned  inside  a  custom-made  wooden  frame  that  kept  their  body  orientation  fixed  and  allowed  them 
to  be  more  easily  positioned  on  the  table.  The  table  was  positioned  so  that  a  region  around  the  liver  would  be  visible  to 
the  biplane  fluoroscopy  system  in  the  room  (Neurostar,  Siemens  Medical  Solutions,  Malvern,  PA,  USA),  and  the 
Aurora  field  generator  was  clamped  to  a  positioning  arm  and  positioned  over  the  abdomen  of  the  animal. 

The  first  step  of  the  study  was  to  register  the  internal  Aurora  coordinate  system  to  ‘animal-centric’  coordinates  (left- 
right,  superior-inferior,  anterior-posterior).  This  registration  was  accomplished  by  placing  a  calibration  frame  on  a 
separate  positioning  arm  just  above  the  animal’s  chest,  as  shown  in  Figure  2.  The  frame  is  a  piece  of  plastic  with 
several  holes  drilled  into  it.  These  holes  have  a  known  spacing  and  orientation.  One  axis  of  the  frame  is  aligned  with 
the  animal’s  superior- inferior  (head-tail)  axis,  and  the  frame  is  leveled  so  that  the  plane  formed  by  the  superior-inferior 
and  left-right  axes  is  parallel  to  the  table  surface.  An  electromagnetically  tracked  needle  is  then  placed  into  each  of  the 
holes  of  the  calibration  frame  and  its  position/orientation  is  recorded  for  10  seconds.  Once  readings  have  been  obtained 
from  all  of  the  holes  in  the  frame,  we  have  sufficient  data  to  calculate  an  affine  transformation  between  the  two 
coordinate  systems. 
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Figure  2:  Aurora  electromagnetic  tracking  system  and  calibration  frame  positioned  over  the  animal.  The  frame  has  several  holes 
drilled  into  it  with  known  spacing.  The  electromagnetically  tracked  needle  can  be  placed  into  the  holes  in  order  to  register  the  Aurora 
coordinate  system  (illustrated  by  the  black  arrows)  to  an  animal-centric  coordinate  system  (illustrated  by  the  white  arrows). 

Once  the  calibration  step  is  complete,  the  electromagnetically  tracked  needle  is  percutaneously  implanted  in  the  liver. 
The  needle  position  is  confirmed  using  biplane  fluoroscopy.  One  or  more  guidewires  are  then  taped  to  the  skin  surface. 
The  guidewire  position  was  not  specified  by  anatomical  landmarks;  rather,  we  tried  to  place  it  on  regions  of  the  skin  that 
showed  the  most  motion.  Likewise,  the  specific  location  in  the  liver  for  the  needle  was  not  specified,  but  we  always 
confirmed  that  it  was  in  the  liver  tissue  and  not  a  large  blood  vessel.  The  setup  for  recording  respiratory  motion  is 
shown  in  Figure  3. 


Figure  3:  Aurora  electromagnetic  tracking  system  set  up  for  respiratory  tracking.  The  needle  is  inserted  percutaneously  into  the  liver, 
and  the  guidewire  is  taped  to  the  skin. 


We  then  recorded  data  from  the  needle  and  guidewire(s)  for  10  to  20  minutes.  During  the  study,  the  breathing  rate 
varied  slightly  from  animal  to  animal  but  was  generally  around  1 1  breaths  per  minute. 
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3.  DATA  ANALYSIS 

Log  files  from  the  Aurora  were  processed  using  Excel  or  Matlab.  Respiratory  motion  data  were  first  transformed  into 
animal-centric  coordinates  using  the  data  from  the  calibration  frame  measurements.  Because  the  Aurora  reads  each 
sensor  sequentially  rather  than  in  parallel,  we  interpolated  the  data  so  that  the  internal  needle  and  any  other  guidewires 
had  the  same  timebase  as  the  first  external  guidewire.  This  gave  us  a  series  of  (x,y,z)  coordinates  for  each  time  point. 
Plots  of  internal  versus  external  coordinates  were  created  and  used  to  determine  the  level  of  correlation  between 
different  axes  of  motion  (for  example,  externa!  left-right  motion  and  internal  left-right  motion). 

We  also  examined  the  hypothesis  that  the  position  of  the  external  guidewire  could  reliably  predict  the  position  of  the 
internal  needle  using  a  linear  model.  First,  we  used  the  entire  data  set  to  obtain  linear  fits  between  internal  and  external 
axes  of  motion  (for  example:  m  *  external  left-right  +  b  =  internal  left-right).  Such  an  equation  essentially  represents  a 
theoretically  optimal  predictor,  because  it  is  based  on  all  of  the  data  from  a  given  animal.  Using  these  equations,  we 
attempted  to  predict  internal  motion  based  on  external  motion  and  evaluated  the  results.  Next,  we  chose  a  small  subset 
of  the  data  by  selecting  either  a  short  time  segment  (25  seconds)  from  the  beginning  of  the  recording  or  a  short  time 
segment  from  another  random  location  within  the  recording.  This  approach  reflects  a  more  realistic  predictor;  in 
radiotherapy/radiosurgery  situations,  a  model  of  respiratory  motion  might  be  calculated  from  an  initial  observation  of 
the  patient’s  breathing  and  then  used  to  carry  out  the  remainder  of  the  treatment.  Data  from  this  subset  was  again 
parameterized  using  a  linear  fitting,  and  the  fitting  coefficients  were  applied  to  the  external  guidewire  coordinates  to 
generate  predicted  internal  needle  positions.  These  predicted  positions  were  then  compared  to  the  actual  recorded 
positions  from  the  experiment  by  calculating  the  root-mean-square  value  of  ( predicted  -  actual),  which  is  defined  as 


RMS-- 


)/Xl (predicted  -  actual )p 

Tn 


where  n  is  the  number  of  compared  values. 


4.  RESULTS 

Figure  4  shows  a  nine-way  correlation  plot  from  one  of  the  experimental  animals.  Each  subplot  represents  the 
correlation  between  one  external  axis  (the  abscissa)  and  one  internal  axis  (the  ordinate),  so  this  represents  an  overview 
of  the  correlation  between  external  and  internal  motion.  In  this  figure,  the  strongest  correlation  (r=0.938)  is  observed 
between  external  posterior-anterior  motion  and  internal  inferior-superior  motion  (bottom  row  middle).  By  contrast,  the 
weakest  correlation  (r=0.485)  is  between  external  left-right  and  internal  posterior-anterior  motion  (top  row  right).  In 
general,  the  external  posterior-anterior  motion  is  the  largest  component  of  external  motion  and  seems  to  have  the  most 
significant  effect  on  internal  position,  whereas  the  effects  of  external  left-right  motion  and  external  inferior-superior 
motion  are  relatively  small. 

These  results  agree  well  with  previously  published  data  from  fluoroscopy  studies9,13  and  with  our  own  anecdotal 
experience  with  respiratory  motion.  Diaphragmatic  movement  in  the  inferior  direction  is  usually  accompanied  by  an 
expansion  of  the  abdominal  region,  so  these  results  are  also  physiologically  sensible. 

Using  the  data  shown  in  Figure  4,  we  calculated  linear  fits  for  each  of  the  subplots.  For  example,  the  bottom  row 
middle  subplot  yields: 

internal  inferior-superior  =  (-0.4993)(extemal  posterior-anterior)-28.95  fit  based  on  all  data 

internal  inferior-superior  =  (-0.5218)(extemal  posterior-anterior)-30.56  fit  based  on  first  25  seconds 

internal  inferior-superior  =  (-0.5066)(extemal  posterior-anterior)-29.45  fit  based  on  random  25  second  segment 

Figures  5  and  6  show  plots  of  the  actual  internal  inferior-superior  motion  vs.  the  predicted  internal  inferior-superior 
motion.  Since  the  correlation  from  the  bottom  row  middle  subplot  was  high,  the  prediction  ability  in  these  cases  is  very 
good,  and  prediction  based  on  a  short  segment  of  the  data  is  nearly  as  good  as  prediction  based  on  the  entire  data  set. 
Active  steering  of  a  radiation  beam  based  on  these  data  should  be  quite  successful. 
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By  contrast,  if  we  repeat  this  fitting  on  the  top  row  right  subplot,  we  obtain: 

internal  posterior-anterior  =  (-0.7440)(extemal  left-right)-50.96  fit  based  on  all  data 

internal  posterior-anterior  =  (-0.5 198)(extemal  left-right)-61 .96  fit  based  on  first  25  seconds 

internal  posterior-anterior  =  (-0.8045)(extemal  left-right)-47.93  fit  based  on  random  25  second  segment 

Figure  7  shows  the  actual  internal  posterior-anterior  motion  versus  predicted  internal  posterior-anterior  motion.  In  this 

case,  the  correlation  from  the  top  row  right  subplot  was  poor,  so  it  is  not  surprising  that  the  prediction  ability  is  also 
poor.  However,  because  the  magnitude  of  internal  posterior-anterior  motion  is  much  less  than  the  magnitude  of  internal 
inferior-superior  motion,  the  RMS  error  of  (predicted-actual)  is  still  relatively  small.  This  result  has  an  interesting  and 
practical  implication  for  respiratory  motion  compensation  using  active  steering:  if  the  motion  along  a  particular  axis  is 
small,  our  inability  to  exactly  predict  that  motion  may  not  have  a  significant  effect  on  the  final  radiation  dose 
distribution. 
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Figure  4:  Plot  showing  the  overall  relationship  between  external  motion  and  internal  motion.  Each  subplot  displays  the  correlation 
between  one  external  axis  and  one  internal  axis.  The  scale  for  all  axes  is  in  mm. 
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Figure  5:  Plot  illustrating  prediction  of  internal  I-S  motion  based  on  all  data.  Solid  line  represents  the  actual  recorded  internal 
motion,  open  circles  represent  the  predicted  internal  motion  based  on  external  skin  data  The  RMS  error  of  (predicted  -  actual)  is 
0.3053  mm. 


Figure  6:  Plot  illustrating  prediction  of  internal  1-S  motion  based  on  the  first  25  seconds  of  the  data.  Solid  line  represents  the  actual 
recorded  internal  motion,  open  circles  represent  the  predicted  internal  motion  based  on  external  skin  data.  The  RMS  error  of 
(predicted  -  actual)  is  0.3280  mm. 


Figure  7:  Plot  illustrating  prediction  of  internal  P-A  motion  based  on  all  correlation  data.  Solid  line  represents  the  actual  recorded 
internal  motion,  open  circles  represent  the  predicted  internal  motion  based  on  external  skin  data.  The  prediction  here  is  clearly  not  as 
good  as  the  previous  two  figures.  However,  since  the  overall  P-A  motion  is  much  smaller  than  the  overall  I-S  motion,  the  RMS  error 
of  (predicted  -  actual)  is  still  quite  small  at  0.3038  mm. 
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5.  CONCLUSIONS 

To  our  knowledge,  this  is  the  first  study  that  has  used  electromagnetic  tracking  to  simultaneously  monitor  internal  and 
external  motion,  and  the  first  study  that  has  examined  the  predictive  ability  of  skin  motion  based  on  such  data. 
Electromagnetic  tracking  offers  key  advantages  in  this  type  of  study.  First,  it  has  the  capability  to  monitor  both  internal 
and  external  motion,  unlike  optical  tracking.  Second,  by  using  the  electromagnetic  tracking  system  for  both  motions, 
we  eliminate  sources  of  error  that  stem  from  needing  to  synchronize  or  register  multiple  separate  tracking  or  imaging 
systems.  Only  a  coordinate  transformation  is  necessary  to  map  the  Aurora  coordinates  into  a  more  natural  coordinate 
system  for  analyzing  breathing  motion.  Third,  electromagnetic  tracking  uses  no  ionizing  radiation,  and  therefore  can  be 
operated  over  a  large  number  of  breathing  cycles.  Finally,  electromagnetic  tracking  is  not  adversely  affected  by 
intervening  layers  of  tissue  and  does  not  require  a  clear  line-of-sight  to  operate.  This  last  property  could  eventually 
provide  a  means  to  move  beyond  predictive  models  of  internal  motion  to  tracking  internal  targets  directly,  which  would 
further  improve  the  accuracy  of  respiratory  motion  compensation  through  active  beam  steering. 

Based  on  the  results  of  this  study,  we  conclude  that  it  is  possible  to  accurately  model  a  portion  of  internal  organ  motion 
by  tracking  the  position  of  markers  on  the  external  skin,  at  least  in  the  experimental  animals  studied  here.  The  accuracy 
of  this  prediction,  computed  as  the  RMS  of  (predicted  -  actual)  is  less  than  1  mm.  Although  this  study  looked  at  the 
motions  by  comparing  single  axes  against  each  other,  we  are  currently  developing  a  fully  three-dimensional  method  for 
studying  the  correlation  and  predictability  in  a  more  comprehensive  way. 

It  is,  of  course,  important  to  recognize  some  inherent  limitations  to  this  study.  First  and  foremost  is  that  the  animals  in 
this  study  were  under  general  anesthesia  and  were  mechanically  ventilated.  Thus,  their  breathing  patterns  were  quite 
regular  and  not  the  same  as  if  they  were  breathing  naturally.  In  some  ways,  this  test  therefore  represents  a  ‘best-case’ 
scenario  because  some  of  the  irregularity  and  complexity  of  normal  breathing  was  removed  by  the  ventilator.  Second, 
given  that  the  study  required  inserting  needles  into  the  liver,  it  may  be  difficult  to  transition  this  technique  into  human 
patients,  especially  normal  healthy  volunteers.  However,  continuous  advancement  in  the  miniaturization  of 
electromagnetically  trackable  tools  and  probes  leads  us  to  believe  that  we  will  eventually  be  able  to  move  this 
technology  into  a  clinical  trial  in  a  radiotherapy  or  radiosurgery  environment. 
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Respiratory  Motion  Compensation  Studies 
Using  a  3D  Robotic  Motion  Simulator  and 
Optical/Electromagnetic  Tracking  Technologies 
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Abstract—  Respiratory  motion  can  degrade  the  quality  of 
nuclear  medicine  images,  especially  when  attempting  to  identify 
small  abnormalities,  make  quantitative  estimates  of  activity 
concentration,  or  track  the  time-varying  location  of  a  tumor. 
Thus,  we  are  developing  methods  for  respiratory  motion 
compensation  and  testing  these  methods  using  robotic  devices. 
The  testing  device  is  a  computer  controlled  3-axis  motion 
simulator  that  can  hold  activity-filled  phantoms  or  spheres  and 
move  them  along  pre-programmed  paths  to  simulate  respiratory 
motion.  The  motion  of  the  platform  is  programmed  in  advance 
and  can  also  be  monitored  using  an  optical  tracking  system,  thus 
providing  a  solid  ground  truth  for  the  time-dependent  activity 
concentration.  Registration  between  the  coordinate  space  of  the 
PET  or  SPET  scanner  and  the  optical  tracker  coordinate  system 
is  based  on  a  set  of  common  points  (mapped  out  using  the  motion 
simulator)  that  are  visible  to  both  systems.  We  have  also  used 
electromagnetic  tracking  and  optical  tracking  to  obtain  realistic 
respiratory  motion  data  from  patients  and  animal  models. 
These  data  can  be  transformed  into  motion  simulator  paths,  thus 
providing  us  with  breathing  patterns  that  accurately  reflect  the 
nonstationary  and  variable  nature  of  human  respiration.  The 
simulator  thus  provides  a  highly  useful  tool  for  repeatably 
testing  different  approaches  to  motion-compensated  image 
reconstruction  or  gated  acquisition  schemes. 

i.  Introduction 

UNCTIONAL  imaging  studies  such  PET  and  SPET  have 
many  uses  in  diagnosing  and  staging  human  disease. 
However,  respiration  can  be  a  severely  complicating  factor  in 
these  studies,  as  many  organs  move  and  deform  during 
normal  breathing.  Given  that  most  PET  and  SPET  studies 
require  the  patient  to  be  imaged  for  approximately  30  to  60 
minutes,  the  organs  of  interest  cannot  be  considered  to  be 
stationary  or  even  to  be  moving  in  a  perfectly  periodic 
fashion.  Furthermore,  the  magnitude  of  respiratory-induced 
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motion  is  typically  larger  than  the  resolution  of  the  detector. 
The  combination  of  these  factors  can  alter  the  apparent 
volume  and  activity  concentration  of  an  abnormality  and  may 
even  render  small  objects  invisible  against  the  noisy 
background  found  in  many  nuclear  medicine  studies. 

Since  patients  obviously  cannot  hold  their  breath  for  the 
entire  length  of  the  study,  it  is  imperative  to  develop  methods 
for  respiratory  motion  compensation.  Although  a  simple 
respiratory  gating  approach  (analogous  to  ECG  gating  in 
cardiac  nuclear  medicine)  could  be  attempted,  this  would 
likely  increase  the  noise  in  the  image  since  the  detector  counts 
would  have  to  be  split  into  several  bins  through  the 
respiratory  cycle.  A  better  approach  would  be  to 
synchronously  record  the  time-varying  respiratory  motion  of 
the  patient  with  the  events  in  the  detector,  and  to  use  all  of  the 
resulting  data  for  the  reconstruction. 

Our  research  group  has  therefore  developed  a  3D  robotic 
motion  simulator  system  that  can  be  used  in  the  development 
and  verification  of  reconstruction  algorithms  that  include 
motion  compensation.  Whereas  patients  typically  have  highly 
variable  breathing  patterns  that  cannot  be  predicted,  the 
simulator  hardware  can  be  programmed  in  advance,  thus 
providing  a  known  time-varying  activity  distribution. 
Respiratory  motion  paths  in  the  simulator  can  be  purely 
periodic  to  provide  simple  test  cases,  or  can  reflect  a  more 
realistic  variable  breathing  pattern.  This  device  thus  fulfills 
two  important  purposes.  First,  it  gives  us  a  platform  for 
testing  the  acquisition  methods  for  respiratory  compensated 
PET/SPET  (e.g.  what  type  of  tracking  system  should  be  used, 
and  what  spatial/temporal  accuracy  is  necessary).  Second,  it 
provides  a  ‘ground  truth’  for  experimental  testing  of 
reconstruction  methods  that  use  motion  compensation;  since 
we  know  the  activity  concentration  as  a  function  of  time,  we 
can  more  readily  make  rigorous,  quantitative  assessments  of 
the  reconstruction. 

II.  Engineering  Design  of  the  Simulator 

The  motion  simulator  is  a  3-axis  motion  platform  using 
orthogonal  linear  slides  (Velmex,  Bloomfield,  NY)  linked  to 
lead  screws  and  motors  (Maxon,  Burlingame,  CA),  as  shown 
in  Figure  1.  Arms  constructed  of  aluminum,  Delrin,  and 
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plexiglass  can  be  attached  to  one  of  the  slides,  which  allows  a 
test  pattern  or  phantom  to  be  moved  within  the  field  of  view 
of  a  PET  or  SPET  gantry.  The  robot  is  controlled  by  a  PC 
through  an  Ethernet-addressable  motion  controller  board 
(Galil,  Rocklin,  CA)  which  drives  the  motors  and  monitors 
their  position  using  feedback  from  rotary  encoders.  Accuracy 
tests  of  a  previous  version  of  the  simulator  showed  that 
individual  axis  error  was  ±0.03  to  ±0.08  mm,  with  an  overall 
system  accuracy  of  ±  0.1  mm  [1],  The  maximum  speed  of  the 
slides  is  several  cm/second,  which  is  far  in  excess  of  the 
speeds  of  typical  respiratory  motion.  Move  commands  are 
processed  every  64  or  128  msec,  so  the  motion  appears 
continuous.  The  range  of  motion  of  the  slides  varies 
depending  on  the  particular  axis  configuration,  but  is  at  least 
6  cm,  which  allows  for  the  full  range  of  respiratory  motion  as 
reported  in  the  literature  [2-3], 

The  simulator  can  be  given  a  periodic  (sinusoidal)  motion 
path  in  one,  two,  or  three  axes  using  a  combination  of  Matlab 
and  C++  programs  written  with  the  Galil  WSDK  (Windows 
Software  Developers  Kit).  Amplitude,  frequency,  and  phase 
can  be  set  to  any  value  within  the  range  of  motion  of  the 
slides  and  the  motor’s  maximum  speed.  We  have  also 
developed  display  tools  for  playing  back  motion  commands 
so  that  they  can  be  easily  visualized.  A  complex  periodic  3D 
motion  path  built  from  three  orthogonal  sinusoidal  paths  is 
shown  in  Figure  2. 

The  simulator  axes  can  be  registered  to  the  coordinate 
space  of  an  imaging  scanner  (such  as  a  PET  or  SPET  system) 
by  attaching  a  radioactive  marker  (as  illustrated  in  Figure  3) 
to  the  arm  and  running  a  calibration  program  that  moves  each 
axis  independently,  thus  mapping  out  3  known  lines  in  the 
coordinate  system  of  the  simulator.  The  resultant  emission 
image  can  then  be  analyzed  to  determine  the  orientation  of 
these  lines  in  imaging  space,  which  allows  us  to  derive  a 
transformation  between  the  scanner  coordinate  system  and  the 
respiratory  motion  simulator  coordinate  system.  We  have 
previously  used  the  simulator  in  this  way  to  imitate  breathing 
motion  artifacts  in  SPET  [4]. 

III.  Acquisition  of  Realistic  Respiratory  Motion  Data 

An  important  characteristic  of  respiratory  motion  is  that  it 
is  highly  variable  and  can  change  over  long  and  short 
timescales.  Furthermore,  accurate  prediction  of  breathing 
patterns  from  a  priori  anatomical  data  such  as  a  static  CT  or 
MRI  scan  is  extremely  challenging.  Thus,  we  have  devoted 
much  of  our  recent  efforts  into  building  up  a  database  of 
recorded  respiratory  motion  and  robot  command  files  —  our 
interface  software  can  be  given  a  contour  of  arbitrary  3D 
points  in  time  and  will  reproduce  that  path  as  closely  as 
possible.  These  data  provide  for  realistic  yet  repeatable 
studies  over  a  wide  range  of  respiratory  conditions,  and  will 
hopefully  serve  as  the  starting  point  for  a  larger  public 
repository  of  dynamic  or  “4D”  images.  We  are  currently 


making  use  of  both  electromagnetic  and  optical  tracking  to 
gather  these  data.  All  studies  were  performed  under  approved 
human  or  animal  research  protocols. 

A.  Electromagnetic  Tracking 

Figure  4  shows  the  Aurora  (Northern  Digital,  Waterloo, 
ON)  electromagnetic  tracking  system  set  up  for  recording 
respiratory  motion  in  a  swine  animal  model.  Electromagnetic 
tracking  provides  a  method  for  continuous  tracking  of 
needles,  guidewires,  and  other  implantable  markers/probes. 
A  field  generator  (the  white  tetrahedron-shaped  object) 
creates  an  oscillating  magnetic  field,  which  induces  current  in 
pickup  coils  in  the  probes.  These  currents  are  then  used  to 
compute  a  position  and  orientation  for  the  probe.  Magnetic 
tracking  has  the  key  advantage  of  being  able  to  track  through 
solid  objects  (which  allows  monitoring  of  internal  organs) 
and  is  capable  of  recording  at  >  10  Hz  over  long  periods  of 
time,  thus  enabling  us  to  observe  high-frequency  periodicity 
and  low-frequency  shifts  in  breathing  patterns,  such  as 
anatomical  settling  or  drift. 

B.  Optical  Tracking 

We  have  recorded  skin  surface  motion  from  patients  using 
both  passive  (Polaris,  Northern  Digital,  Waterloo,  ON)  and 
active  (Flashpoint,  Image  Guided  Technologies,  Boulder,  CO; 
Optotrak,  Northern  Digital,  Waterloo,  ON)  optical  tracking 
systems.  These  studies  were  first  reported  in  [5]  and  are 
continuing  as  an  integral  part  of  Cyberknife  radiosurgery 
treatments  at  our  institution.  Optical  tracking  is  completely 
non-invasive,  has  sub-millimeter  position  accuracy,  can  track 
continuously  over  long  periods  of  time,  and  can  operate  over 
3-4  meters  range  as  long  as  line-of-sight  is  preserved. 
Typical  sampling  rates  with  optical  tracking  are  >  20  Hz, 
which  is  more  than  sufficient  for  measuring  respiratory 
motion.  Active  markers  can  be  tracked  more  reliably  than 
passive  markers,  but  are  more  expensive  and  are  not  currently 
available  in  wireless  form.  Figure  5  shows  a  sample  Polaris 
recording  from  a  patient,  demonstrating  a  somewhat  steady- 
state  breathing  pattern  with  intermittent  deviations.  We  are 
currently  seeking  institutional  approval  to  begin  using  this 
tracking  system  on  patients  scheduled  for  PET  and  SPET 
scans. 

IV.  Conclusions 

The  respiratory  motion  simulator  provides  a  means  for 
quantitatively  examining  the  effects  of  patient  movement 
during  radionuclide  imaging.  A  key  advantage  of  the  system 
is  its  flexibility;  it  can  be  programmed  to  generate  motion 
paths  of  any  size  and  shape.  This  flexibility  allows  us  to 
easily  simulate  different  types  of  motion  (breathing,  cardiac, 
patient  fidgeting,  etc...)  so  that  we  can  study  their  effects. 
Furthermore,  while  the  system  has  great  flexibility,  its  motion 
is  also  highly  repeatable  due  to  the  precision  of  the  motors 
and  sliders.  We  are  gathering  data  on  realistic  respiratory 
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motion  patterns  and  creating  an  excellent  test  bed  for 
evaluating  respiratory  motion  compensation  methods  and 
reconstruction  algorithms. 

The  simulator  is  therefore  a  unique  and  valuable  tool  for 
the  development  and  testing  of  methods  to  compensate  for 
motion-induced  blurring  in  PET  and  SPET.  We  are  currently 
implementing  optical  tracking  of  skin  motion  in  our  SPET 
and  PET  scanners,  and  will  use  this  tracking  with  the  motion 
simulator  to  help  develop  reconstruction  algorithms  that 
compensate  for  respiratory  motion.  We  are  also  investigating 
“4D”  PET  and  SPET  imaging  for  determining  the  time- 
dependent  position  of  a  “hotspot”  on  functional  imaging. 
Such  information  will  be  vital  in  the  development  of 
interventional  radiology  and  radiosurgery  therapies  that 
include  motion  compensation,  as  motion  data  will  enable  a 
new  level  of  precision  for  image-guided  therapy. 


Fig.  1.  Picture  of  the  3D  robotic  motion  simulator.  The  straight  extension 
arm  extends  out  from  the  linear  stages  to  allow  a  test  pattern  or  phantom  to  be 
placed  inside  the  active  area  of  the  scanner. 


Fig.  2.  Sample  complex  path  for  the  3D  respiratory  motion  simulator.  Axis 
units  are  in  mm,  with  the  longest  axis  (range  of -10  to  10)  representing  the 
inferior-superior  axis  on  the  patient.  By  adjusting  phase,  frequency,  and 
amplitude,  both  simple  and  complex  periodic  motion  paths  can  be  obtained. 


Fig  3.  Motion  simulator  arm  shown  with  radioactivity-filled  spheres  attached 
for  calibration,  which  allows  the  simulator  coordinate  space  to  be  matched  to 
the  imaging  system  coordinate  space. 
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Fig.  4.  Setup  for  electromagnetic  tracking  experiments  in  swine  model.  The 
white  tetrahedron  is  the  magnetic  field  generator.  Induced  currents  are 
measured  from  a  coil  at  the  end  of  the  needle  (which  has  been  implanted  into 
the  liver)  and  a  guidewire  (taped  to  the  skin  surface  just  to  the  right  of  the 
needle  insertion  point). 


Fig  5.  Respiratory  motion  from  skin  recorded  by  Polaris  optical  tracking 
system.  Short-term  periodicity  and  larger,  intermittent  jumps  in  the  position 
are  both  evident  in  this  recording. 
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Abstract.  The  motion  of  internal  organ  targets  represents  a  major  challenge  for  image-guided 
therapy.  However,  in  many  clinical  situations,  the  internal  organ  motion  cannot  be  observed 
directly,  so  it  is  necessary  to  find  a  surrogate  signal  that  is  both  easily  observable  in  the  therapy 
setting  and  predictive  of  internal  organ  motion.  We  are  therefore  studying  the  relationship 
between  skin  motion  and  internal  organ  motion  using  dynamic  magnetic  resonance  imaging 
(MRI),  in  order  to  determine  the  feasibility  of  using  an  MRI-based  model  to  perform  respiratory 
motion  compensation  in  image-guided  therapies.  In  this  paper,  we  report  (1)  testing  of  different 
dynamic  MRI  pulse  sequences  and  which  ones  appear  most  suitable  for  respiratory  motion 
modeling,  and  (2)  results  from  quantitative  analysis  of  skin  and  liver  motion.  This  study  will 
provide  the  groundwork  for  future  experiments  that  will  directly  test  the  validity  of  using  skin 
motion  to  predict  complex  internal  target  motions. 

Keywords:  Magnetic  Resonance  Imaging;  motion  tracking;  respiration;  image-guided  therapy 


1.  Introduction 

Modem  interventional  radiology  techniques,  coupled  with  advanced  therapeutic 
methods  (e.g.  radiofrequency  ablation,  high  frequency  ultrasound)  have  enabled  a  new 
generation  of  image-guided  therapies.  Similarly,  the  advent  of  intensity  modulated 
radiotherapy  (IMRT)  and  highly  adaptable  radiosurgery  accelerators  such  as  the 
Cyberknife®  (Accuray,  Sunnyvale,  CA)  have  increased  the  accuracy  and  precision  of 
radiation  medicine  treatments.  A  crucial  challenge  for  future  image-guided  interventions 
is  the  ability  to  either  monitor  or  predict  the  position  of  internal  organ  targets  during  a 
procedure.  This  ability  is  especially  important  for  procedures  in  the  thorax  and 
abdomen,  where  respiration  creates  significant  involuntary  organ  motion. 
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In  an  ideal  situation,  one  would  monitor  the  internal  organ  target  directly  and  steer  the 
therapy  based  on  that  data.  Although  some  technologies  such  as  fluoroscopy  or 
interventional  MRI  allow  for  the  direct  monitoring  of  internal  targets  during  therapy, 
these  technologies  are  not  feasible  for  every  image-guided  procedure  and  also  have 
significant  disadvantages.  For  example,  fluoroscopy  is  typically  limited  to  two- 
dimensional  projections,  and  necessitates  lengthy  exposures  to  ionizing  radiation. 
Interventional  MRI  requires  an  extensive  infrastructure  of  highly  specialized  and  costly 
equipment  and  cannot  be  implemented  in  a  radiotherapy  environment  because  of 
magnetic  and  radiofrequency  interference.  Thus,  since  our  ability  to  directly  observe  the 
internal  target  is  limited,  we  must  instead  develop  methods  for  predicting  its  position. 

Skin  motion  is  mechanically  linked  to  the  movement  of  the  underlying  rib  cage, 
diaphragm,  and  abdominal  muscles,  and  thus  it  should  in  principle  be  related  to  the 
motion  of  the  organs  in  the  torso.  Existing  studies  of  this  relationship  using  fiducial 
markers  and  fluoroscopy  have  shown  encouraging  results  [1-2].  Furthermore,  skin 
motion  can  be  readily  monitored  using  mechanical  devices  (e.g.  bellows,  stretch 
transducers)  or  optical/infrared  tracking  cameras  such  as  the  Polaris®  (Northern  Digital, 
Waterloo,  Canada)  system.  For  these  reasons,  our  laboratory  is  developing  methods  for 
measuring,  analyzing,  and  simulating  respiratory-induced  skin  and  organ  motion,  with  a 
focus  on  applications  in  interventional  radiology  and  Cyberknife  radiosurgical  therapies. 
This  particular  project  focuses  on  developing  suitable  dynamic  MRI  techniques  for 
imaging/modeling  the  simultaneous  respiratory  motion  of  the  liver  and  the  skin  surface. 
Ultimately,  we  plan  to  use  these  data  to  determine  if  a  surrogate  signal  —  motion  of  the 
skin  —  can  be  predictive  of  internal  target  motion. 

2.  Methods 

Volunteers  for  the  study  were  recruited  under  an  IRB-approved  human  subjects 
protocol.  This  protocol  recruited  only  normal  patients  with  no  history  of  liver  disease, 
since  we  were  focusing  on  the  liver  as  the  internal  organ  of  interest.  Volunteers  were 
scanned  in  either  a  1.5T  General  Electric  MRI  system  in  the  Department  of  Radiology  at 
Georgetown  University  Hospital/Medstar  Health  or  a  3T  Siemens  MRI  system  at  the 
Georgetown  University  Center  for  Functional  and  Molecular  Imaging.  Patients  were 
positioned  on  the  imaging  table  and  a  respiratory  bellows  was  placed  around  their  thorax 
or  abdomen  to  monitor  their  breathing.  The  1.5  T  system  used  a  surface  coil  as  the 
receiver,  whereas  the  3T  system  used  the  general-purpose  body  coil  as  the  receiver  since 
no  surface  coils  were  available. 

We  tested  four  potential  MR  pulse  sequences:  (a)  T1  weighted  in-  and  out-of-phase 
images,  (b)  T2  weighted  half-Fourier  single  shot  turbo  spin-echo  (HASTE),  (c)  T2/T1 
weighted  fast  imaging  with  steady-state  precession  (trueFISP),  and  (d)  3D  fast- 
acquisition  multi-echo  (FAME). 

Breath-hold  coronal  and  axial  images  were  acquired  at  end-inspiration  and  end- 
expiration  using  both  T1  and  T2  weighted  sequences.  Once  the  breath-hold  images  were 
completed,  coronal  and  axial  images  were  acquired  under  free  breathing,  again  using 
both  T1  and  T2  weighted  sequences.  For  some  patients,  we  tested  prospective 
respiratory  gating  based  on  the  respiratory  bellows  to  ensure  that  the  images  were 
derived  from  known  points  in  the  respiratory  cycle.  We  also  tested  prospective 
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respiratory  gating  using  navigator  pulses  centered  on  the  liver-right  lung  border.  The 
navigator  pulse  is  essentially  a  very  fast  ID  MRI  image  along  the  inferior-superior  axis; 
it  determines  the  respiratory  phase  by  monitoring  the  movement  of  the  diaphragm. 

Images  from  the  different  MRI  pulse  sequences  were  evaluated  subjectively  for  noise, 
contrast,  and  the  presence  of  motion  artefacts.  Although  the  goal  of  the  MRI  study  is  not 
to  produce  images  for  disease  diagnosis,  the  images  should  still  have  sufficient  detail  to 
make  out  organ  borders,  the  skin  surface,  and  key  internal  structures. 

Breath-hold  and  prospectively  gated  images  were  analyzed  by  a  semi-automatic 
routine  for  extracting  the  outline  of  the  skin  from  each  axial  slice  and  then  combining 
these  outlines  into  a  surface  map.  The  outline  of  the  liver  was  obtained  through  manual 
segmentation  to  create  a  volume.  The  liver  position  and  orientation  were  compared 
between  end-inspiration  and  end-expiration,  and  changes  in  the  skin  position  were  also 
measured. 

The  non-gated  images  acquired  under  free  breathing  (trueFISP)  were  first  pre- 
processed  to  sort  the  files  according  to  their  acquisition  time,  so  that  we  could  bin  the 
slices  into  their  approximate  location  in  the  respiratory  cycle.  This  binning  generated  4- 
6  image  sets  reflecting  the  anatomy  at  different  time  points.  Analysis  of  these  images 
then  proceeded  as  with  the  breath-hold  images. 

3.  Results 

For  breath-hold  images,  the  T1  weighted  in/out-of-phase  images  generally  produced 
the  clearest  contrast  of  internal  organ  boundaries  since  this  pulse  sequence  is  specifically 
designed  to  accentuate  soft  tissue  borders.  Flowever,  attempting  this  sequence  under  free 
breathing  produced  significant  motion  artefacts.  The  3D  FAME  sequence  provides  more 
isotropic  spatial  resolution  than  the  other  sequences,  but  it  was  also  severely  affected  by 
respiratory  motion  blurring  since  it  required  phase  encoding  in  two  directions. 

The  T2  weighted  HASTE  and  the  T2/T1  weighted  trueFISP  sequence  could  both  be 
applied  under  free  breathing.  Respiratory  gating  was  not  available  for  the  trueFISP 
sequence,  so  these  images  were  rebinned  manually.  For  the  T2  weighted  HASTE 
sequence,  prospectively  gated  images  using  navigator  pulses  had  very  similar  image 
quality  to  those  taken  with  breath  holding,  and  the  protocol  was  much  easier  on  the 
patients.  Sample  T2  weighted  HASTE  images  are  shown  in  Figure  1. 


Fig.  1 .  Comparison  of  T2  weighted  HASTE  images.  Image  on  the  left  is  acquired  during  a  breath  hold, 
whereas  the  image  on  the  right  is  acquired  using  navigator  gating  during  free  breathing. 
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Images  from  the  trueFISP  sequence  produced  good  soft  tissue  contrast  and  were 
generally  less  noisy  than  the  HASTE  images.  Sample  T2/T1  weighted  trueFISP  images 
are  shown  in  Figure  2. 


Fig.  2.  Coronal  T2/T1  weighted  trueFISP  images  of  the  liver  (different  coronal  planes)  acquired  during  free 
breathing.  Respiratory  gating  was  not  available  with  the  trueFISP  sequence,  so  the  images  were  acquired  at 
subsecond  time  intervals  and  then  manually  binned  into  different  points  of  the  respiratory  cycle. 

The  breath-hold  images  are  useful  because  they  show  the  maximum  range  of  internal 
organ  motion  and  thereby  provide  a  "baseline"  for  comparison  against  the  images  taken 
during  normal  breathing.  However,  the  range  of  motion  seen  between  end-inspiration 
and  end-expiration  tends  to  be  larger  than  that  seen  under  normal  breathing,  since 
patients  usually  inhale  quite  deeply  before  the  breath  hold.  Conversely,  prospective 
respiratory  gating  is  much  less  difficult  for  the  research  subjects,  but  also  is  less  of  a 
‘snapshot’  of  the  anatomy  because  each  slice  is  acquired  during  a  separate  respiratory 
cycle.  This  lengthens  the  total  imaging  time  and  also  can  create  slice  mismatches  if  the 
anatomy  does  not  return  to  the  exact  same  position  during  each  breathing  cycle. 

The  semi-automated  routine  developed  for  determining  the  outline  of  the  skin  is 
illustrated  in  Figure  3.  Rays  are  drawn  on  axial  slices,  and  the  skin  surface  is  identified 
by  detecting  large  changes  in  the  pixel  value  along  the  raypath.  The  ensemble  of  these 
points  outlines  the  skin  surface  on  that  slice,  and  these  outlines  are  then  combined  to 
form  a  surface  map  of  the  skin. 


Fig.  3:  Illustration  of  semi-automated  method  for  detecting  skin  surface,  showing  an  anterior-posterior  raypath 
across  the  image  and  the  intersection  of  the  raypath  and  the  skin  surface. 
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This  method  was  generally  reliable,  and  only  required  manual  editing  for  about  1-2%  of 
the  detected  skin  surface  points. 

Manual  segmentation  of  the  liver  was  generally  straightforward,  although  occasional 
guidance  from  a  radiologist  was  helpful  when  the  border  became  indistinct.  A  sample 
liver  volume  is  shown  in  Figure  4. 


Fig.  4.  Surface  rendered  model  of  the  liver  derived  from  manually  segmented  MRI  data. 

Movement  of  the  skin  varied  depending  on  the  individual  and  the  anatomical  location, 
and  ranged  between  0  and  1.6  cm.  Liver  motion  was  most  significant  in  the  cranio- 
caudal  direction;  displacements  as  large  as  4.4  cm  were  observed  in  breath-hold  images, 
whereas  under  free  breathing  the  largest  displacement  was  3.2  cm. 

4.  Discussion 

MRI  can  be  a  valuable  tool  for  exploring  the  respiratory-induced  movement  of  organs 
and  skin.  Because  no  ionizing  radiation  is  used,  it  is  possible  to  perform  this  study  on 
volunteers  with  very  low  risk  and  to  image  them  over  many  respiratory  cycles,  which  is 
essential  for  testing  the  repeatability  and  reliability  of  matching  skin  motion  to  internal 
organ  motion.  Specific  absorption  rate  (SAR)  limitations  are  naturally  an  important 
concern,  especially  when  imaging  in  a  3T  magnet,  but  can  often  be  addressed  by 
changing  the  parameters  of  the  pulse  sequence,  such  as  the  flip  angle. 

This  study  extends  the  results  of  previous  MRI  studies  of  liver  motion  secondary  to 
respiration.  Rohlfing  et  al.  [3]  investigated  liver  motion  modeling  using  MRI  imaging 
with  prospective  gating,  but  found  that  the  gating  created  impractically  long  imaging 
times.  Conversely,  Blackall  [4]  employed  a  protocol  where  volunteers  were  asked  to 
hold  their  breath  at  different  points  in  the  respiratory  cycle  (e.g.  end-inspiration,  end- 
expiration,  midway  through  the  cycle,  etc.)  but  reported  that  patients  could  not  reliably 
hold  their  breath  at  arbitrary  points  in  mid-cycle;  the  majority  of  the  breath  holds  were 
very  similar  to  end-exhalation.  Neither  approach  was  therefore  optimal.  However,  since 
our  ultimate  goal  is  to  track  and  treat  a  moving  internal  target  while  the  patient  breathes, 
the  gating  approach  is  more  reflective  of  the  clinical  task.  Fortunately,  current  state-of- 
the-art  MRI  technology  has  enabled  significant  decreases  in  the  imaging  time,  so  that  a 
single  prospectively  gated  set  of  MRI  images  can  be  acquired  in  less  than  three  minutes. 

Nevertheless,  there  are  areas  for  potential  improvement  in  the  future.  The  first  is  that 
the  2D  MRI  sequences  image  only  one  slice  at  a  time,  so  even  if  the  slices  can  be  read 


presented  in  Computer  Assisted  Radiology  and  Surgery  (CARS)  2004 


out  in  rapid  succession,  they  are  not  truly  simultaneous  in  the  way  that  slices  from  a 
multirow  CT  scan  would  be.  However,  the  advent  of  multichannel  receiver  coils  and 
parallel  imaging  techniques  should  help  to  address  this  issue.  We  expect  to  install  an 
eight-channel  torso  coil  in  the  very  near  future,  which  should  improve  both  speed  and 
image  quality. 

The  second  potential  area  for  improvement  is  in  the  gating.  Although  the  prospective 
gating  works  well,  a  more  efficient  use  of  the  imaging  time  would  be  to  record  the 
respiratory  phase  (either  with  respiratory  bellows  or  continuous  navigator  imaging) 
during  trueFISP  imaging  and  use  those  data  to  retrospectively  gate  the  images  to 
different  parts  of  the  respiratory  cycle.  The  current  software  does  not  allow  us  to 
separately  record  the  raw  data  from  the  respiratory  bellows  or  to  continuously  record 
navigator  images,  but  we  hope  to  develop  such  techniques  during  the  coming  months. 
We  are  also  investigating  the  potential  for  retrospectively  gated  3D  MRI  sequences. 

5.  Conclusion 

MRI  is  a  very  useful  too!  for  simultaneously  observing  skin  and  internal  organ  motion. 
We  have  tested  several  different  imaging  sequences,  identified  ones  that  will  be  well- 
suited  to  the  task  of  modeling  liver  motion,  and  developed  methods  for  quantitative 
analysis  of  the  MRI  images.  In  the  next  phase  of  this  project,  we  will  examine  the 
relationship  between  skin  motion  and  liver  motion  in  greater  detail,  in  order  to  test  the 
hypothesis  that  the  skin  motion  can  be  used  to  predict  the  motion  of  internal  targets, 
thereby  allowing  us  to  track  and  treat  them  effectively. 


Acknowledgements 

This  work  was  funded  by  the  U.S.  Army  grant  DAMD 17-99- 1-9022.  The  content  of 

this  manuscript  does  not  necessarily  reflect  the  position  or  policy  of  the  U.S. 

Government.  The  authors  would  like  to  thank  Bemd  Kuhn,  PhD,  of  Siemens  Medical 

Solutions,  Erlangen,  Germany,  for  his  advice  on  MR  pulse  sequences  and  Elliot  Levy, 

MD,  of  Georgetown  University  Hospital/Medstar  Health  for  guidance  with  liver 

segmentation. 

References 

[1]  Schweikard  A,  Glosser  G,  Bodduluri  M,  Murphy  MJ,  Adler  JR.  Robotic  motion  compensation  for 
respiratory  movement  during  radiosurgery.  Comput  Aided  Surg  2000;  5(4),  263-277. 

[2]  Ozhasoglu  C,  Murphy  MJ.  Issues  in  respiratory  Motion  Compensation  during  External  Beam 
Radiotherapy.  Int.  J.  Radiation  Oncology  Biol.  Phys.  2002;  52(5):  1389- 1399. 

[3]  Rohlfing  T,  Maurer  Jr  CR,  O'Dell  WG,  Zhong  J.  Modeling  liver  motion  and  deformation  during  the 
respiratory  cycle  using  intensity-based  nonrigid  registration  of  gated  MR  images.  Medical  Physics  2004; 
31(3),  427-432. 

[4]  Blackall,  JM.  Respiratory  Motion  in  Image-Guided  Interventions  of  the  Liver.  Doctoral  Thesis,  King's 
College,  London,  2002. 

[5]  Clifford  MA,  Banovac  F,  Levy  E,  Cleary  K.  Assessment  of  hepatic  motion  secondary  to  respiration  for 
computer  assisted  interventions.  Comput  Aided  Surg  2002;  7(5),  291-299. 


appendix  10 


presented  in  Computer  Assisted  Radiology  and  Surgery  (CARS)  2004 


Electromagnetically  tracked  guidewires  for 
interventional  procedures 


2  a  b  b  •  c 

Bradford  J.  Wood  ’  ,  Dave  Lindisch  ,  Sohan  Ranjan  ,  Neil  Glossop  , 

Kevin  Clearyb 

“Clinical  Center,  National  Institutes  of  Health,  Bethesda,  MD,  USA 
4 Imaging  Science  and  Information  Systems  Center,  Department  of  Radiology, 
Georgetown  University  Medical  Center,  Washington,  DC,  USA 
cTraxtal  Technologies,  LLC,  Bellaire,  TX,  USA 


1.  Introduction 

We  have  prototyped  and  tested  an  electromagnetically  tracked  guidewire  for 
interventional  procedures.  Electromagnetic  tracking  of  instruments  in  the  interventional 
suite  enables  image  overlay  of  preoperative  CT  scans  and  provides  multimodality 
guidance  while  decreasing  x-ray  dose.  An  electromagnetically  tracked  guidewire  was 
manipulated  in  a  swine  model  with  co-display  of  CT  and  angiographic  images. 

2.  Methods  and  Results 

To  evaluate  electromagnetic  tracking  (AURORA™,  Northern  Digital,  Inc.)  in  the 
interventional  suite,  we  developed  an  image-guided  surgery  system.  The  system  was 
developed  using  open  source  software  components  including  FLTK  for  the  user 
interface,  VTK  for  visualization,  and  ITK  for  image  processing. 

The  study  was  done  under  an  approved  animal  care  protocol.  An  enhanced  CT  scan  of 
a  swine  chest  and  abdomen  was  obtained.  A  rigid  body  registration  was  carried  out  by 
identifying  the  points  on  the  CT  scan  and  in  electromagnetic  space.  The  guidewire  was 
then  initially  moved  in  free  air  over  the  abdominal  area  of  the  swine  to  verify  that  the 
registration  software  was  working  and  that  is  could  be  tracked. 

The  internal  jugular  vein  and  right  carotid  artery  were  percutaneously  accessed  using 
ultrasound  guidance  and  modified  Seldinger  technique  with  a  21  gauge  needle  and  an 
.018  inch  guidewire.  As  the  guidewire  was  advanced  down  the  aorta,  the  corresponding 
axial  CT  scan  was  displayed  simultaneously  with  the  real-time  angiographic  x-ray 
image.  The  system  provided  good  qualitative  agreement  in  the  opinion  of  the 
experienced  interventional  radiologist  manipulating  the  guidewire.  As  the  guidewire  was 
advanced  into  the  selected  vessel,  the  image-guided  system  was  able  to  actively  track  the 
guidewire  and  display  axial,  coronal,  and  sagittal  CT  scans  at  the  same  level,  using  the 
electromagnetically  tracked  sensor  position  to  select  the  slice  location. 
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Abstract.  The  purpose  of  this  study  was  to  develop  a  respiratory  motion  simulator  and  evaluate  its 
accuracy.  The  simulator  is  intended  for  use  in  the  CyberKnife  radiosurgery  suite  for  phantom 
treatments,  dose  verifications,  and  future  technical  developments.  CyberKnife  is  a  radiation 
treatment  system  that  incorporates  a  robotic  arm  to  precisely  position  a  linear  accelerator.  For 
enhanced  accuracy  in  the  treatments  of  a  tumor  in  an  internal  organ  that  is  subject  to  a  patient’s 
respiratory  motion,  the  CyberKnife  robot  must  be  commanded  to  move  the  radiation  beam  in  real¬ 
time  to  compensate  for  such  motion.  The  motion  simulator  described  in  this  study  was  specifically 
developed  to  serve  the  CyberKnife  technical  requirements  for  better  treatment  accuracies.  The 
motion  simulator  consists  of  two  platforms  that  may  be  programmed  independently  to  reproduce 
different  three-dimensional  respiratory  motions  at  both  the  skin  over  the  patient’s  abdomen  and  at 
the  internal  tumor  site.  The  simulator  meets  the  spatial  restrictions  of  the  CyberKnife  suite  with  a 
position  accuracy  better  than  ±0. 1  mm. 

Keywords:  respiratory  motion,  robotic  motion  simulator,  stereotactic  radiosurgery,  CyberKnife 


1.  Introduction 

The  CyberKnife®  stereotactic  radiosurgery  system  (Accuray,  Inc.)  incorporates  a 
6  MeV  linear  accelerator  mounted  on  a  robotic  arm  to  deliver  multiple  crossfire  beams 
of  radiation  to  destroy  a  lesion  while  sparing  surrounding  healthy  tissue  and  critical 
structures  [1,  2].  In  addition,  this  system  can  correct  for  patient  movement  during 
treatment  through  the  use  of  skeletal  landmarks  or  small  implanted  fiducials.  In  August 
2001,  CyberKnife  was  cleared  by  the  Food  and  Drug  Administration  for  tumor 
treatments  anywhere  in  the  body,  and  since  that  date,  thousands  of  treatments  have  been 
completed  successfully. 

About  85%  of  current  CyberKnife  treatments  are  for  brain  and  spine  tumors,  which  do 
not  move  significantly  with  patients’  respiration.  However,  for  many  internal  organs, 
respiratory  motion  does  cause  significant  movement  of  the  tumors,  such  as  lung  tumors 
as  reported  in  [3],  To  address  this  issue,  Accuray  developed  the  Synchrony  technology 
with  the  goal  of  compensating  for  tumor  motion  due  to  respiration. 
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With  Synchrony,  a  correlation  model  between  the  skin  motion  and  a  given  internal 
organ  motion  is  established  by  simultaneously  taking  x-ray  images  to  track  internal 
fiducials  and  using  infrared  trackers  attached  to  the  skin.  This  correlation  model  is 
established  and  verified  during  treatment.  The  CyberKnife  robot  follows  the  internal 
organ  motion  based  on  the  prediction  from  the  skin  motion  [4].  To  quantify  and  validate 
this  technology,  a  programmable  respiratory  motion  simulator  is  required.  The  simulator 
must  precisely  simulate  the  three-dimensional  respiratory  motion  at  both  the  skin  on  the 
patient’s  abdomen  and  at  the  tumor  site  in  the  internal  organ.  This  paper  presents  our 
simulator  design  and  initial  accuracy  experiments. 

Figure  1  shows  the  CyberKnife  Suite  in  the  Department  of  Radiation  Medicine  at 
Georgetown  University  Hospital.  The  robot  arm  has  a  full  six  degrees  of  freedom.  Two 
orthogonal  x-ray  cameras  with  corresponding  amorphous  silicon  detectors  allow  on-line 
verification  of  the  patient’s  position  for  necessary  correction.  A  fixed  theoretical  point, 
named  iso-center,  is  located  in  the  middle  of  the  cross  section  of  the  two  x-ray  beams. 
During  treatment,  the  targeted  tumor  is  located  at  the  iso-center  through  which  all  the 
radiation  treatment  beams  pass.  Figure  1(c)  shows  this  center  represented  by  the  tip  of  a 
calibration  post. 


(a)  (b)  (c) 


Fig.  1.  CyberKnife  Suite  at  Georgetown  University  Hospital 

2.  Methods 

The  respiratory  motion  simulator  (RMS)  consists  of  two  independent,  programmable 
robotic  three-dimensional  (3-D)  motion  platforms.  The  platforms  are  called  a  Skin- 
Motion  Simulator  (SMS)  and  a  Tumor-Motion  Simulator  (TMS).  The  RMS  was  built 
completely  with  off-the-shelf  components.  Wiring,  assembly  and  software  development 
were  completed  in-house  at  Georgetown  University  Medical  Center.  The  components 
were  selected  based  on  the  technical  requirements  of  motion  amplitude  and  frequency, 
which  were  established  from  real  patient  data  taken  at  Georgetown  University  Hospital 
(SMS)  [5]  and  published  data  (TMS)  [3],  External  loads,  as  well  as  the  weight  of  the 
components  themselves,  were  also  considered  in  the  motor  calculation. 
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The  SMS  consists  of  three  high-precision  orthogonally-mounted  linear  slides  (Velmex, 
Inc.),  and  the  TMS  consists  of  two  such  linear  slides  and  one  vertically-mounted  linear 
piston  actuator  (Ultra  Motion,  LLC).  Each  stage  includes  a  high-performance  servo 
motor,  an  integrated  gearhead  at  the  motor  output  shaft,  and  an  optical  rotary  encoder 
mounted  on  the  motor  back  shaft  (Maxon  Precision  Motors,  Inc.).  The  platforms  are 
controlled  by  a  PC  through  an  Ethernet-addressable  motion  control  board  (Galil  Motion 
Control,  Inc.)  that  utilizes  feedback  from  the  encoders. 

A  major  challenge  in  designing  the  simulator  was  the  unique  geometry  requirements 
of  the  CyberKnife  system.  As  mentioned  in  Section  1,  the  CyberKnife  uses  two 
orthogonal  x-ray  beams  to  track  internal  fiducials.  Therefore,  only  radio-lucent  materials 
(i.e.,  non-metallic  materials)  may  present  in  the  paths  of  these  x-ray  beams.  This  no¬ 
metal  zone  limits  the  size  of  slides  and  motors  that  may  be  used.  In  addition,  the  iso¬ 
center  is  located  directly  in  the  middle  of  this  no-metal  zone  where  phantoms  or  film  box 
for  dose  verification  must  be  located.  Considerations  must  also  be  taken  to  ensure 
collision-free  movements  between  the  CyberKnife  robot  and  the  simulator,  as  well  as 
between  the  two  platforms.  To  correctly  select  and  design  the  components  of  the 
simulator,  a  3-D  computer-aided  design  (CAD)  method  was  adopted  using  the  software 
Alibre  Design  (Alibre,  Inc.).  Figure  2  shows  the  design  results  from  a  CAD  model,  and 
the  actual  simulator  in  the  CyberKnife  Suite. 


Fig.  2.  Respiratory  Motion  Simulator 


3.  In-situ  verification  and  accuracy  tests 

After  assembly,  the  simulator  was  tested  to  verify  its  capability  at  in-situ  condition  as 
well  as  its  accuracy  in  reproducing  both  sinusoidal  waves  and  human  respiratory 
motions.  The  accuracy  tests  were  conducted  in  the  Computer  Integrated  Surgical 
Systems  and  Technology  (CISST)  Lab  at  Johns  Hopkins  University  using  an  Optotrak  to 
evaluate  the  simulated  motions.  The  Optotrak  (Northern  Digital,  Inc.)  is  an  optical  high- 
precision  3-D  motion  measurement  system  that  can  achieve  a  resolution  of  0.01  mm  and 
an  accuracy  of  0.1  mm  (root-mean-square  or  rms),  both  at  a  2.25  m  distance  from  the 
object  (http://www.ndigital.com/optotrak.html).  This  is  sufficient  for  the  qualification 
purpose  of  the  simulator  since  only  the  relative  position  movement  is  concerned. 
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3. 1  In-situ  tests  for  dimension  and  dose  verification 

The  motion  simulator  was  tested  in  the  CyberKnife  Suite  at  Georgetown  University 
Hospital  under  radiosurgery  treatment  conditions.  The  simulator  was  placed  on  the 
treatment  table  with  the  electronics  placed  on  the  floor  about  2  m  away  from  the  iso¬ 
center  to  prevent  the  6MeV  x-ray  beams  from  passing  directly  through  any  electronic 
components  (see  Figure  2b).  The  RMS  was  linked  through  a  30  m  Ethernet  cable  to  a  PC 
outside  of  the  treatment  vault.  A  film  cube  with  two  GaFchromic  films  was  placed 
perpendicularly  on  the  TMS  top  platform.  Six  metal  fiducial  markers  in  the  film  cube 
were  used  to  track  its  position.  The  TMS  was  then  moved  in  sinusoidal  motions  with  7.5 
mm  amplitude  in  the  craniocaudal  direction  and  5  mm  in  both  the  anterior-posterior  and 
the  left-right  directions. 

It  has  been  verified  that  the  RMS  fits  the  physical  space  and  no  collision  will  occur 
between  the  CyberKnife  robot  and  the  RMS,  or  between  the  SMS  and  the  TMS.  The 
CyberKnife  x-ray  image  system  confirmed  that  no  metal  parts  interfered  with  fiducial 
tracking  of  the  film  cube  on  the  TMS  platform.  Ionizing  radiation  did  not  interfere  with 
the  operation  of  the  RMS. 

The  RMS  was  also  used  to  demonstrate  the  distortions  of  the  dose  distribution  in  a 
tumor  under  uncompensated  breathing  motion.  Three  treatments  with  a  maximum  dose 
of  3000  cGy  were  delivered.  For  the  first  treatment,  the  TMS  was  not  moved,  so  the 
ideal  dose  distribution  was  obtained  (Figure  3a).  For  the  second  treatment,  the  TMS  was 
driven  by  the  sinusoidal  motions,  but  no  position  corrections  were  performed.  The 
CyberKnife  treatment  was  applied  as  if  the  film  block  would  have  remained  in  the  initial 
position.  This  dose  distribution  would  simulate  an  IMRT  treatment  under  free  breathing 
conditions.  Treatment  two  showed  an  elongated  dose  distribution  in  the  direction  of  the 
largest  motion  amplitude  (Figure  3b).  In  addition,  the  peak  dose  was  smeared  out.  The 
center  of  the  distribution  was  also  offset,  which  demonstrated  the  respiratory  motion 
effect  relative  to  a  treatment  plan  done  on  a  breath-hold  CT  scan.  For  the  third  treatment, 
two  x-ray  images  were  taken  before  every  radiation  beam  for  position  correction.  This 
treatment  also  showed  a  smeared  out  dose  distribution,  but  the  distribution  was  more 
circular  than  for  the  second  treatment  (Figure  3c).  It  should  be  noted  there  is  a  significant 
time  delay  between  taking  the  x-ray  images  and  firing  the  treatment  beams,  so  position 
correction  based  on  x-ray  images  cannot  completely  compensate  the  respiratory  motion 
in  real  time. 


Fig.  3.  Digitized  films  of  radiation  dose  distribution 
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3.2  Accuracy  evaluation  tests 

To  verify  the  accuracy  of  the  RMS,  tests  were  carried  out  using  the  Optotrak,  which 
was  placed  about  2  m  from  the  simulator.  For  each  test,  two  sets  of  data  were  collected: 
one  set  measured  the  motion  at  the  top  stage  of  the  simulator  in  Optotrak’s  coordinates, 
the  other  measured  the  commanded  motion  to  each  slide  in  the  simulator’s  coordinates. 
A  registration  method  was  applied  to  convert  the  Optotrak  results  into  the  simulator’s 
coordinates  for  comparison.  Figure  4  shows  the  system  setup  of  the  accuracy  tests. 

First,  the  simulator  was  driven  with  sinusoidal  waves  at  individual  stages,  which 
resulted  in  1-D  motions  along  each  axis  of  the  simulator.  These  sinusoidal  waves  show  a 
frequency  of  0.2  Hz  and  peak-to-peak  amplitudes  ranging  from  7.5  mm  to  15  mm.  Next, 
the  simulator  was  tested  with  the  same  waves  at  all  stages  simultaneously,  thus  providing 
a  3-D  motion  at  the  top  stage.  Finally,  a  segment  of  about  60  seconds  of  real  human  3-D 
respiratory  motion  recorded  at  the  chest  skin  was  used  to  drive  both  the  SMS  and  the 
TMS.  This  set  of  data  contains  about  eight  breathing  cycles  with  an  average  amplitude  of 
10  mm  and  one  deep  breath  of  17  mm  in  total  (see  Figure  5). 


Fig.  4.  Test  setup  for  RMS  accuracy  evaluation  Fig.  5.  Human  3-D  respiratory  motions 


The  comparisons  show  that  both  the  SMS  and  the  TMS  can  achieve  an  accuracy  of 
±0.1  mm  in  simulating  both  sinusoidal  waves  and  human  respiratory  motion.  This 
accuracy  is  a  combination  of  all  three  axes.  For  each  individual  axis,  the  accuracy  is 
even  better,  ranging  from  ±0.03  mm  to  ±0.08  mm  (e.g.,  Figure  6).  Random  errors  of 
±0.02  mm  were  detected  even  when  the  simulators  remained  motionless.  This  is  due 
primarily  to  the  resolution  of  the  Optotrak  system  and  possibly  to  some  nearby 
disturbances  that  caused  slight  vibration  in  the  system  setup. 

It  is  evident  the  position  error  depends  heavily  on  motion  amplitude  and  frequency; 
that  is,  larger  motion  amplitude  or  higher  frequency  are  predictably  correlated  with 
larger  position  error.  Moreover,  the  total  error  is  normally  a  superposition  of  two  distinct 
frequencies  -  a  low  frequency  part  equivalent  to  the  nominal  motion  frequency,  and  one 
or  more  much  higher  frequency  components  that  correspond  to  the  natural  frequencies  of 
the  simulator  (see  Figure  6a).  This  is  due  to  dynamic  forces  during  the  movement  that 
cause  deflection  and  vibration  in  the  structure.  Therefore,  structural  stiffness  should  be 
improved  further  for  better  accuracy. 
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Time  (sec.) 

Fig.  6.  Sample  results  of  accuracy  tests 


4.  Conclusions 


In  this  study,  a  robotic  RMS  has  been  developed.  It  consists  of  two  platforms,  namely 
SMS  and  TMS,  and  both  platforms  may  be  programmed  independently  to  simulate 
different  3-D  motions.  It  has  been  demonstrated  that  the  RMS  is  able  to  reproduce 
human  respiratory  skin  and  tumor  motions  with  a  precision  better  than  ±0.1  mm.  The 
RMS  fits  the  physical  space  and  no-metal  zone  requirements  for  the  CyberKnife  Suite. 
The  basic  hardware  design  fulfilled  our  expectation.  Further  improvements,  e.g.,  better 
structural  stiffness  for  less  vibration  and  user-friendly  software  interface,  will  be 
conducted.  We  anticipate  the  simulator  will  be  a  valuable  tool  in  the  CyberKnife  Suite 
for  phantom  treatments,  dose  verifications  of  Synchrony  technology  and  future  technical 
developments. 
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